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EXECUTIVE SUMMARY 



In the third year of the Writing Enhancement Program, the Florida 
legislature has mandated that all senior high school students in grades ten 
through twelve be served by the program which has as its goal the 
improvement of writing in secondary schools. As part of this legislation , 
funding has been allocated to reduce class size so that teachers have more 
time to evaluate students 1 writing production. In 1983-84 ari evaluation of 
the Writing Enhancement Program was conducted to determine if the 
program was implemented according to guidelines and if the writing 
production of students improved as a result of participation in the Writing 
Enhancement Program, An analytic scoring procedure was used to measure 
student progress in writing in four areas: composition, sentence 
formation, usage, and mechanics. 

Conclusions for 1983-84 

1) In general, the program was effective in improving stu- 
dents' writing. Students in 78 percent (11 of 14) of senior 
high course levels improved in some aspect of their writing 
skills • 

2) By course level: Basic students, especially those in grades 
10 and 11, showed the most improvement overall, 

3) By grade level: eleventh grade students, especially those 
in basic classes and advanced classes showed improvement 
in the most areas* These two course levels also showed the 
most improvement during 1982-83, 

4) Students in three course levels, tenth grade regular, tenth 
grade advanced and twelfth 'grade AP, showed no improve- 
ment in any assessed area. Based on their pre-scores, 
twelfth grade AP students may already have possessed 
reasonable skills in all assessed areas, 

5) Most student growth occurred in two areas: composing 
and/or sentence formation. Whenever students significantly 
improved in the composing trait, their overall writing ability 
also improved significantly. 
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0.0 INTRODUCTION 

In the third year of the Writing Enhancement Program, the Florida 
legislature has mandated that all senior high school students grades ten 
through twelve be served by the program which has as its goal the im- 
provement of writing in secondary schools. As part of this legislation, 
funding has been allocated to reduce class size so that teachers have more 
time to evaluate students 1 writing production. Last year all juniors and 
seniors were served by the Writing Enhancement Program; for 1983-84 the 
program has been expanded to include sophomores as well. In Hillsbo- 
rough County program guidelines have been prepared and disseminated to 
all high schools. 

Last year (1982-83), evaluation of the Writing Enhancement Program 
included both process and product data. Interviews and record audits 
indicated that the program was being implemented according to program 
guidelines and was viewed by most participants as being successful. 
Product data were collected in the form of pre/ post- writing samples from a 
random sampling of students at each course level which was part of the 
Writing Enhancement Program. A holistic scoring procedure was used in 
that each writing sample was given one score based on its overall merit. 
Students in four of the six course levels evaluated showed significant 
improvement in their overall writing abilities . Compensatory Education 
students' writing was evaluated separately; overall, they did not signifi- 
cantly improve in their writing production . Subsequent feature analysis 
revealed some specific characteristics of students 1 writing and ways in 
which their writing improved. 
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The purpose of the 1983-84 evaluation of the Writing Enhancement 
Program was to determine if the program was implemer ^ording to 

guidelines and if the writing production of students impro. id as a result 
of participation in the Writing Enhancement Program, Specific questions 
included the following: 

1) Were paragraphs or essays written by high school 
students at the end of a vear's instruction better than 
paragraphs or essays written bv the same students at 
the beginning of the year? 

2) In what specific ways did the writing of Writing En- 
hancement Program students improve after involvement 
in the writing program? 

3) V/ere county guidelines for instrv >tion and record- 
keeping for the Writing Enhancement Program imple- 
mented by participating teachers? 

4) What unmet needs in the area of writing instruction 
exist at the high school level? 

In response to questions 1 and 2V y program developers and senior high- 
school teachers involved in the Writing Enhancement Program suggested 
that an a^iytic scoring procedure be used to measure student progress in 
writing. Unli.ke the holistic method which assigns an overall score to the 
writing pieces, analytic scoring is designed to investigate particular char- 
acteristics of a piece of writing, such as organization or mechanics. 
Teachers thought that this more specific evaluative information would be 
helpful in identifving program strengths and weaknesses. This document 
will report pretest-to-posttest changes in composition performance for 
fourteen course levels. The remainder of the report is presented in five 
subsections : Program Description , Evaluation Design , Procedures , 
Analyses / Results , and Conclusions / Recommendations . 



.0 PROGRAM DESCRIPTION 



In recent years, the State of Florida placed greater emphasis on the 
development of writing skills at the secondary level and allocated increased 
funding to local districts to implement a writing program. During the 
1981-82 school year, the first year of funding, implementation of the 
program in Hillsborough County was left primarily to individual schools. 

It quickly became apparent that some standard of uniformity was 
needed throughout the county to assure meeting state guidelines while 
covering county material. A workshop devoted entirely to developing a 
writing enhancement curriculum was held during the summer of 1982; the 
Hillsborough County Writing Enhancement Program was the result of that 
workshop . 

The local program's objective is to focus attention on developing 
competent writing skills within Hillsborough County classrooms by institut- 
ing uniform content and proficiency standards for grade and ability levels. 
The program guide ( Writing Enhancement Program , Hillsborough County 
Public Schools , 1982 ) is a functional teaching guide emphasizing writing 
skills and accompanied by models and alternative classroom assignments. 
The suggested curriculum presupposes the interaction of writing with all 
facets of English. It focuses attention on the sequencing of skills. 
Skeletal in nature, it encourages application across teaching styles, 
literary modes and other English elements while providing sufficient guide- 
lines for implementing a writing enhancement program. It emphasizes the 
elements of grammar, literature, and writing as unique but interrelated 
components of written communication. 



Students in grades 10, 11 and 12 participate in the Writing Enhance- 
ment Program through their English classes. In conjunction with other 
secondary English course objectives, students are instructed in the identi- 
fication, romprehension , and utilization of effective writing skills. Accord- 
ing to Writing Enhancement Program curriculum guidelines, each student is 
required to complete at least nine documented writing assignments per 
grading period. Student folders are maintained as evidence of sequential 
program delivery. Method(s) of evaluation of student writing assignments 
is (are) at the discretion of the classroom teacher. 
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EVALUATION DESIGN 



This section will describe the evaluation questions and subordinate 
investigation that were addressed by the evaluation and will describe the 
subjects and instruments that were used* 

Because insti\ictional emphasis and expectation varies with student 
level , the district elected to conduct two distinct studies, one for each of 
two student population types , One study required that performance 
changes in paragraph writing be examined, and the other required that 
perfon mce changes in essay writing be examined. The evaluation ques- 
tions were the same for each study, however. 

As^ previously stated, the following evaluation questions were ad- 
dressed by the study: 

1) Are paragraphs /essays written by high school students 
at the end of a year's instruction better than para- 
graphs or essays written by the same students at the 
beginning of the year? 

2) In what specific ways does the writing of Writing 
Enhancement Program students improve after involve- 
ment in the program? . 

Because several types ot students are served by the Writing Enhance- 
ment Program, the district wished to investigate whether different results 
may be associated with different types of students. In addition to the 
primary evaluation questions given above , therefore , the following addi- 
tional question was also investigated: 



. Is there differential performance across the groups listed 
below? 

PARAGRAPH : 

Compensatory Students 

Basic Students 

Regular Students (grade 10) 

ESSAY : 

Regular Students (grade 11 and 12) 
Advanced Students 
Advance Placement Students 

To address both the primary evaluation questions and the subordinate 
investigation, writing samples were collected from students representing 
each of the groups identified above. A sampling plan was developed to 
ensure representation of the student groups named in Table 1, below. 



Table 1 

Writing Enhancement Program Subjects 





Grade 10 


Grade 11 


Grade IZ 


Compensatory 


P 


P 


P 


Basic 


P 


P 


P 


Regular 


P 


r 


E 


Advanced 


E 


E 


E 


Advance Placement 


E 


E 


E 



P = Paragraph 



E = Essay 

At each of eleven high schools two teachers per course per grade 
were selected to collect writing samples from randomly selected classes of 
students; class s ~es ranged from 15 to 25 students. 

er|c * 6 12 



All students in the selected classes produced a pre-writing sample 

during the week of October 3-7; post-samples were collected May 7-11. 

Based on these procedures, the following numbers of paragraphs and 
* essays were generated: 

Paragraphs : 6 course levels x 25 students per class x 2 
classes pe^ school x 11 schools = 3,300 sample 
pairs (pre/post) ; 

Essays ; 8 course levels x 25 students per class x 2 

classes per school x 11 schools = 4,400 sample 
paris (pre/post) . 



All writing samples were matched by student n^me. Approximately 
ten paired samples were chosen randomly per course level per school for a 
total of 100 pairs per course. Thus, ah at 500* paragraph pairs .and 800 
essay pairs were scored. 

Two types of instruments were used: test instruments and scoring 
instruments. Each type is described in this sub-section. 

Test Instruments . Separate tests were developed for the para- 
graph and for the essay as follows: 

PAF.AGRAPH t Four topics were developed to test para- 
graph writing. These topics were paired to form two 
sets of two topics each. For the pretest, administered 
during the first two weeks of the school year, Set I 
topics were given to half of the students and Set II to 
the other half. For the posttest , administered in May , 
the alternate set was administered to each group of 
students . On both occasions , then , students had a 
choice of two topics. On each occasion, however, stu- 
dents chose only one topic on which to write their 
paragraphs. 

ESSAY : Two topics were developed to test efisay writ- 
ing. On each occasion students were assigned a topic, 
half receiving topic A on the pretest and half receiving 
Topic B . On the posttest , each half was assigned th? 
topic not assigned on the pretest . 
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All topics were developed by Hillsborough County instruc- 
tional personnel, and administration procedures were developed by 
the Department of Testing and Evaluation. Paragraph topics and 
procedures are shown in Appendix A; essay topics and procedures 
are shown in Appendix B. 

S-.oring Instruments . A representative group of teachers from the 
district identified a set of 21 traits to be evaluated in the para- 
graph and a set of 17 traits +o be evaluated in the essay. The 
consultant grouped each set of traits under four domains: Com- 
posing, Sentence Formation, Usage, and Mechanics. Then, 
separately for each study, these domains were defined with special 
tailoring for the traits under examination. When the domains had 
been defined in terms of the identified trait, a four-point scale 
was developed for each domain, for each study. These domain 
score scales provided for each paper receiving four independent 
scores, one in each domain. This domain scoring was designed to 
accommodate the district's need for discrete information to answer 
Evaluation Question 2. 

The domain definitions and score scales were formatted, along 
with background material and general information about proce- 
dures, into the Writing Enhancement Program Scoring Manual . 
This document was the central instrument used to train raters for 
each study. A copy of the Writing Enhancement Program Scoring 
Manual accompanies this report as Attachment 1. 

Results were analyzed using a series of Wilcoxan Ranked-Signs Tests 
for paired samples. In Section 4.0 of this document results are reported 
for each course level. Data for total scores are reported as well as for 
each individual trait examined. 
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As previously noted , the following evaluation questions were also 
addressed by the study: 

3) Were county guidelines for instruction and record- 
keeping for the Writing Enhancement Program imple- 
mented by participating teachers? 

4) What unmet needs in the area of writing instruction 
exist at the high school level? 

All teachers whose students submitted writing samples as part of the 
product evaluation were sent a survey in the spring. The survey con- 
tained questions regarding instruction, record-keeping, and unmet needs. 
Unfortunately only six teachers (@10%) returned this survey. Neither 
evaluators nor program developers felt this response rate was sufficiently 
repiesentative of the English teacher population to draw valid conclusions 
about program operation. Thus, no results ?re reported for these two 
evaluation questions. Follow-up will occur during preplanning for 1984-85 • 



3.0 SCORING PROCEDURES 

Scoring activities were conducted in five sequential stages: Paper 
Preparation, Pre-Scoring, Rater Training, Scoring, and Follow-Up, Each 
stage will be discussed in detail in the following sub-sections. 

3. i Scoring Preparation 

Following initial planning meetings, the consultant prepared a 
logistics document to guide all paper preparation and other scoring- 
related activities. The logistics document provided instructions for 
the following activities for each of the two studies. 

1) Randomly selecting matched pairs of papers from 
each group to be included in the scoring sample. 

2) Recording a seven-digit numerical code on each 
paper, then removing all identifying information 
from the paper. 

3) Scrambling paper order to interrupt all unit 
clusters. 

4) Pre-coding a scannable scoring form for each 
paper . 

5) Preparing scoring packets that, contained a des- 
ignated number of papers and their corresponding 
scoring forms. 

6>) Preparing ancillary materials for the scoring 
session . 

The logistics document also detailed procedures and staff respon- 
sibilities for training, scoring, and follow-up activities. A copy of 
"Scoring Logistics - Hillsborough County Scoring Project - Writing 
Enhancement Program" appears as Appendix C of this report. 
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Pre- Scoring 

It can be claimed, with reason, that anyone who teaches writing 
should know how to "grade" or score the papers that students write. 
Not only should they be able to do so, most are able to do so. 
Nonetheless, teachers often are startled, in discussion with their 
equally well-trained and conscientious peers, to learn that their 
colleagues do not always share their composition priorities. Some 
teachers place greater emphasis on neatness than others. Some are 
more concerned with the structural correctness- of sentences; others 
are more concerned with the richness of language in sentences. Some 
teachers place greater emphasis on a composition's content; others 
place greater emphasis on a composition's form. Although all of these 
priorities are worthy ones, they can be the root of considerable 
disagreement among raters as to the score that a given paper should 
receive, thereby resulting in unreliable and invalid scores. 

To counteract the influences of these conflicting priorities during 
scoring activities, a single standard, established by expert judges 
who represent the district's priorities, is established in advance 
through pre-scoring activities. Pre-scoring is done to establish the 
standard for all ensuing scoring and to ensure the validity both of 
the process and of the scores ultimately given to each paper. During 
pre-scoring, a team of expert judges who represent the system's 
present standards undergo intensive training. They then score, 
discuss, and rescore until they have reached agreement about the 
final score on a large number of papers. These pre-scored papers 
then are used to stabilize the score scale throughout subsequent 
training and scoring activities. 




Some pre-scored papers become Anchor Papers, so called because 
they anchor or define the score scale for training purposes. Raters 
study these papers, along with the score scale, early in the training 
process. Another set of pre-scored papers become Training Papers 
and are used to teach raters to score in concert with the pre-scoring 
standard, rather than apply personal standard of their own. Still 
another set of pre-scored papers become Check Papers . These 
papers are used periodically throughout scoring as a check to ensure 
that raters are not drifting away from the standard set during pre- 
scoring. 

To conduct pre-scoring for this program, 72 paragraph samples 
and 95 essay samples were representatively selected from those sub- 
mitted across all respective groups. These papers were blinded and 
pre-sorted such that all unit clusters were interrupted. Each type 
then was organized for pre-scoring. 

Pre-scoring activities for the paragraph were conducted on June 
14, and for the essay, on June 15. Both sessions were conducted in 
the offices of the Department of Testing and Evaluation. Thr«e 
pre-scoring judges were selected as follows: two classroom teachers, 
and one measurement evaluation%pecialist. The consultant designed 
and directed pre-scoring activities. 

The consultant gave each judge a copy of the W riting Enhance- 
ment Program Scoring Manual and reviewed the manual's introduction 
and the sections related to hand-scoring issues and procedures and 
regulations. Each of the paragraph topics and their administration 
instructions also were reviewed. The consultant then introduced the 
judges to domain-referenced scoring by carefully reviewing the 




section "Expectations for the Composition" which defined each of the 
four composition domains and identified each of the specific traits to 
be examined in the paragraph. The consultant explained to the 
judges that each paper would be assigned four different scores by- 
each judge, one for performance in each of the four domains. The 
consultant further explained that each score was to reflect the stu- 
dent's general performance on all skills identified for the referenced 
domain . 

Once this section had been reviewed, judges studied the score 
scales for each domain. To begin pre-scoring activities, each judge 
was given a set of copies of the. same five papers. Judges were 
asked to read each paper then, one by one, evaluate each domain by 
comparing the paper's quality in each respective domain with descrip- 
tors on that domain's, score scale. Judges used this procedure to 
score each of the five papers, without discussion with other judges 
and without regard for any other paper in the set. 

Once each judge had read and scored all five papers, the con- 
sultant collected all scores. Scores were collected by domain for each 
paper, and all discussion was conducted by domain, rather than by 
paper. First, papers on which all judges had assigned the same 
score (firm-agreement) were identified, one at a time. For each, the 
consultant gave the paper's identification number and the judges' 
score, now designated the "true score" for the domain.' Judges then 
were asked to re-read the paper and the score's domain descriptor to 
firmly establish the rationale for the paper's True Score in that 
domain. Judges briefly summarized the domain's True S^ore rationale 
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on the back of the score card to more firmly establish the scale's 
standard in their minds. 

Once this activity had been completed for all of the firm- 
agreement papers in each domain, judges again reviewed the papers, 
their scores and rationale statements. Judges then were asked to 
re-read the remaining papers and rescore them, utilizing the clarifica- 
tion that had resulted from their review of firm-agreement papers. 
After this second scoring, results were collected and the above 
procedures were repeated for all papers on which there was firm 
agreement. 

Papers which still had discrepant scores were then identified and 
the following procedures were employed to resolve the disagreement. 

1) The consultant identified the scores that judges had 
assigned to the discrepant paper. Usually, only two 
scores were represented in the disagreement. For 
example, two judges may have assigned a score of "3" 
and one judge may have assigned a score of "4." 

2) The consultant then referred judges to firm-agreement 
papers which represented each of the assigned scores 
and judges were asked to compare the discrepant 
paper to these papers. Judges were to determine 
which set of papers the discrepant paper was most 
like. This comparison was followed by a discussion of 
the discrepant paper's similarity to other papers. 
After the discussion, the consultant called for a third 
scoring of the discrepant paper. 

Once the above procedures had been completed for one domain, 
they were repeated for the next domain until true scores had been 
established in each domain for each paper. These iterative proce- 
dures were employed for the first two sets of pre-scoring papers. 
After these sets had been scored, however, iterative scoring was 
utilized only in cases where each judge had assigned a different 
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score, or where one of the assigned scores was not contiguous to the 
other two scores. Otherwise, the consultant resolved discrepant 
scores by defining the True Score as the score assigned by two of 
the three judges. 

To reflect True Score agreement on pre-scored papers, these 
papers were grouped into three categories for each domain. These 
categories and their definitions follow. 

A) Firm-Agreement Papers - pre-scored papers on which 
all judges assign the same score in the domain after 
the first reading. 

B) Iterative-Agreement Papers - pre-scored papers on 
which ail judges assign the same score in a domain 
after two or more readings. 

C) Resolved-Agreement Papers - pre-scored which had 
contiguous discrepancies after one reading and which 
were resolved by the consultant. 

Table 2 displays the proportion of pre-scored paragraphs that 
were grouped into each of the three categories in each domain, as a 
result of pre-scoring. 

Table 2 

Writing Enhancement Program 
Pre-Scored Paragraphs bv^ Agreement Category 





Category 


A . 


B 


C 


Composing 


.29 


.13 


.58 


Sent Form 


.38 


.11 


.51 


Usage 


.50 


.07 


.13 


Mechanics 


.49 


.09 

.... ., ., 


.42 
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The True Score distribution for each domain, for pre-scored 
paragraphs is shown in Table 3. 



Table 3 

Writing Enhancement Program Pre-Scored Paragraphs 
Distribution of True Scores 
N = 72 





4 


3 


2 


1 


Composing 


.06 


.44 


.42 


.08 


Sent Form 


.03 


.50 


.46 


.CI 


Usage 


.01 


.64 


.33 


.01 


Mechanics 


.04 


.67 


.28 


.01 



Procedures used to pre-scor^ essays were identical to those used 
to pre-score paragraphs. Table 4 shows the proportion of pre-scored 
essays that weru grouped inlo each of the agreement categories for 
each domain. 



Table 4 

Writing Enhancement Program Pre-Scored Essays 
" By Agreement Category ~~ 

M a 95 





Categories 


A 


B 


C 


Composing 


.32 


.13 


.55 


Sent Form 


.49 


.09 


.42 


Usage 


.61 


.08 


.32 


Mechanics 


.42 


.12 


.46 
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The True Score distribution for each domain, for pre-scored 
paragraphs is shown in Table 5. 

Table 5 

Writing Enhancement Program P r e-Scored Essays 
Distribution of True"~5cores 
N = 95 





4 


3 


2 


1 


Composing 


.09 


.39 


.49 


.02 


Sent Form 


.09 


.85 


.05 


.00 


Usage 


.09 


.91 


.00 


.00 


Mechanics 


.11 


.84 


.05 


.00 



Rater Training 

To prevent any bias based upon raters 1 knowledge of the levels 
of students who were assigned essays and the levels who were as- 
signed paragraphs , the two composition types were scored separately. 
All training and scoring for the paragraph were conducted first. 
When all paragraphs had been scored, raters were retrained for the 
essay, and all essays then were scored. Although scoring criteria 
differed, all procedures for training and scoring the paragraph and 
the essay were the same. 

To prepare for rater training, the consultant selected from 
among pre-scored papers the Anchor Papers, Training Papers, and 
Check Papers for use in each scoring session. Because training 
emphasis was r'.aced on differentiating the four domains, all Anchor, 
Training, and Check Papers were selected for score distribution and 
agreement category distribution at the domain level. Selection was 
based on the following considerations. 



I7 23 



ANCHOR PAPERS ; Special consideration was given to 
selecting Anchor Papers that illustrated as many score 
combinations and as wide a range of score distribution 
within each domain as possible. 

TRAINING PAPERS AND CHECK PAPERS ; The same 
considerations given to selecting Anchor Papers were given 
to selecting these papers. In addition, care was taken to 
select paper sets that would provide raters with a full 
representation of central scale values (scores of . 2 and 3), 
traditionally difficult values for raters to separate. 

Once each of the above paper types had been selected for para- 
graphs and for essays, sufficient copies of each set were made for 
each rater. Sets were then collated and labeled for use in rater 
training and scoring activities. 

Training and scoring activities were conducted at the Holiday 
Inn, North, in Tampa, on June 18 through 22. Twenty-two raters 
were selected by district staff and were representative of schools 
throughout the district. A list of raters and their school affiliations 
is shown in Appendix D. 

Training activities for the paragraph required approximately six 
hours. Raters were first given a review of the Writing Enhancement 
Program and were given brief information about the make-up of the 
.scoring sample. They were told what grade levels were represented in 
the scoring sample and that both^ pretest and posttest papers were 
represented. The consultant explained to raters that all identifying 
information had been removed from the papers and that the papers 
had been scrambled so that there would be no grade-level, group, or 
occasion clusters in their packets. 

Each rater then was given a copy of the Writin g Enhancement 
Program Scoring Manual and each section of the Manual was carefully 
reviewed with them by the consultant. Raters were given time to 
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review the "Expectations" section, the score scale for each domain , 
and the Anchor Papers. Raters discussed these materials among 
themselves and with the consultant until they felt reasonably comfort- 
able with them. 

Raters then were given papers for Training Round #1 (T-l) . 
Each rater was given a set of the same five papers and a score sheet 
on which to record each paper's four scores. Raters read and scored 
each ^aper independently. They were instructed to utilize the score 
scales in combination with the Anchor Papers to determine the scores 
for each of the papers in T-l. 

Once raters had finished scoring papers in T-l, their score 
sheets were collected and their scores for each papei recorded on the 
consultant's appropriate Domain Monitor Sheet. Once all raters 1 
scores had been transferred to the monitor sheets, each domain was 
examined individually, and raters 1 scores were compared to each 
paper's True Score. Calculations were made to determine the number 
of discrepant scores for each rater (incidence) and whether their 
discrepancies tended to be high or low as compared with the true 
scores (tendency) . In cases where raters 1 Incidence or Tendency 
indicated a problem, this information was related to them and rec- 
ommendations were made to them for adjustment of their scoring. 
Otherwise, their scoring was simply monitored to assess individual 
accuracv . These procedures were repeated for each domain and 
continued throughout all training rounds and check rounds. 

Once each rater's accuracy in each domain had been evaluated, 
raters' score sheets were returned to them along with a self-adhesive 
iabel that displayed the true scores for each paper in the set. 
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Raters were instructed to affix the label to the front cover of the 
paper set and to review these true scores . Where their scores were 
discrepant from the true scores, raters were instructed to utilize 
Anchor Papers and other Training Papers, along with discussion to 
clarify the discrepancy. In cases where as many as one-third of the 
raters were discrepant on a single paper in any domain, the consul- 
tant discussed the paper with the entire group. Raters were not 
allowed to dispute or otherwise challenge the true scores. 

When raters had completed their discussions, they were given 
another set of Training Papers and the procedures described above 
were epeated. Altogether, five Training Rounds were conducted. 
Monitor sheets for each Training Round are displayed in Appendix D. 
These sheets document quite strong agreement among raters through- 
out training activities. Positive and negative Tendency ratings 
indicate whether a Plater was scoring higher or lower than the True 
Score, respectively. The value of the Tendency rating indicates how 
many points above or below the True Score the rater scored for the 
entire round. 

The Incidence value reoresents the number of discrepant scores 
assigned by the rater for the round. An Incidence rating of zero 
indicates that the rater assigned no discrepant scores in that round, 
A rater who had an Incidence rating of 2 or higher, but had a Ten- 
dency Rating of 0, showed discrepancy from the True Scores, but no 
consistent Tendency to overscore or underscore. 
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3 . 4 Scoring 

Each scoring round 'was conducted in exactly the same manner. 
Each rater was given a scoring packet containing a set of papers and 
their corresponding scoring forms. Raters were instructed to first 
check the order of their materials to ensure that they were recording 
information on the correct form for each respective paper. Raters 
then read each paper independently and coded in each' domain score 
and their rater identification number on the appropriate score form. 
When raters had finished scoring the entire packet of papers, the 
packet was collected by a scoring assistant and the ra'ter was given 
another packet of papers to score. Collected packets were checked to 
ensure that each paper had been properly coded in. When each score 
form in a packet had been checked for accuracy and completeness, 
the four scores on each form were covered with a WRITE GUARD 
Rating Shield to obscure them from view by the second rater. All 
papers and score forms then were placed in identical order and 
returned to their packets. 

When scoring assistants collected the second packet from raters, 
they gave raters a packet that already had received a first reading. 
Raters followed their routine rating procedures, this time recording 
their identification numbers and scores on the section of the form 
designated for the second reading. 

When all distributed packets had received two readings, a new 
scoring round began. This procedure continued until all papers had 
been scored by two readers. 

When packets were collected from the second reading, the rating 
shield was removed from each score form and each form was checked 
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-fgr^completeness «nd accuracy. The two scenes for each domain were 
then compared. When any pair of scores was found to be discrepant 
by more than one value (non-contiguous scores) > the discrepant 
domain was checked for resolution, and both sets of scores were 
covered by rating shields. Such score forms were placed on top of 
the ethers and these packets were set aside for resolution scoring . 

After all scoring packets had received two readings , packets 
which contained papers with discrepant domain scores were 
distributed for resolution . Resolution readings were conducted 
exactly as other readings had been, except that only the discrepant 
domain was scored bv the resolution rater. The resolution rater 
recorded score information in the Etrea of the score form designated 
for resolution scores. 

When all scoring activities had been completed, rating shields 

a- 

were removed from score forms that had required resolution, and each 
form was checked carefully for accuracy and completeness. All score 
forms were then cleaned for stray marks, stacked, and returned to the 
district office for scanning. 
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ANALYSIS/RESULTS 



Answers to the first two evaluation questions were obtained after data 
for each course level, each writing trait, and overall ability were analyzed 
with a series of nonparametric statistical tests. As a result of scanning all 
scoring documents and conducting a thorough search for all available data, 
two data files (one for paragraph scores, one for essay scores) were 
established. Then appropriate programs from the Statistical Package for 
Social Sciences ( SPSS ) were employed to analyze existing data. Due to the 
ordinal nature of the rating scale (1-4), the Wilcoxan Matched-Pairs 
Ranked Signs Test was used to compare pre-writing sample scores with 
post-writing sample scores. It was assumed that students in all course 

v 

/ 

levels would improve their writing skills in at least one of the specified 
writing traits and/or in their overall writing ability. 

Writing Enhancement Program stu'dents in eleven of the fourteen 
course levels significantly improved in some aspect of their writing skills. 
Figures 1 through 5 depict average pre- and post-writing sample means, 
growth slopes, and relative score positions for each trait and each course 
level. Accompanying table? (6-10) show numbers of students who experi- 
enced negative, positive, or no no gain from pre- to post-writing samples, 
along with the significance level for each trait. 

.As shown in figure 1 and table 6 # Compensatory Education students 
in grade 10 significantly improved their performance in the composing trait 
and overall writing ability. Grade 11 Compensatory Education students 
significantly improved in the mechanics trait.. 
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.Students in grade 10 basic classes (figure 2 and table 7) showed 
significant improvement in three traits: composing, sentence formation and 
mechanics. In addition, their overall writing ability significantly im- 
proved. Grade 11 basic students also showed significant improvement in 
three traits: sentence formation, usage and mechanics. Their overall 
writing ability also improved. Twelfth grade basic students significantly 
improved in the usage trait. It is important to note that in ^hree out of 
four traits and in overall writing ability, eleventh grade ba3ic students 
scored higher than the other two grade levels on the post-writing sample. 

.Regular classes wrote either paragraphs (grade l(j)) or essays 
(grades 11 and 12) as pre-post assessments. Figure 3 and t£ble 8 indicate 
no improvement by tenth grade students in any trait or in overall writing 
ability. Indeed, in two out of four traits and in overall ability, students 
scored lower on the post-writing paragraph than on the pre -writing para- 
graph sample. Grade 11 regular students significantly impjroved in sen- 
tence formation and overall writing ability. Grade 12 ,reg,ul&r, students 
showed significant improvement in composing and overall writing ability . 

.Results for advanced classes are shown in figure 4 and table 9. 
Once again, tenth grade students showed no improvement in aiiy trait or 
in overall ability; in both composing and ^tence formation mor,e students 
had lower post-scores than pre-scores. i '.<,venth grade students showed 
significant improvement in every trait as wrll as overall writing ability. 
Grade 12 students significantly improved in sentence formation. It is 
important to point out for twelfth grade students that, as a group, they 
scored above the "3" scale point on both pre and post assessments in 
three out of four traits. That is, they begar. the year with at least 
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"reasonable" skills in the assessed areas and maintained these skills during 
the school year. 

• Figure 5 and table 10 show results for advance placement (AP) 
classes. Tenth grade students significantly improved in the composing 
trait and overall writing ability. Grade 11 students significantly improved 
in sentence formation. Grade 12 students showed no significant improve- 
ment in any area. As mentioned previously, these 12th grade students 
began the year with at least reasonable skills in all areas and maintained 
them during the year. Their average pre and post scores in all areas 
except one were above all other groups (except grade 11 AP sentence 
formation post score) . 

• Table 11 provides an overview of all course levels and a listing of 
the areas in which students significantly improved. In terms of course 
type, basic class students improved in the most areas. Eleventh grade 
advanced students improved in all areas: every trait and overall writing 
ability. Students in three course levels, tenth grade regular, tenth grade 
advanced, and twelfth grade AP, showed no improvement in any area. 
Table 12 shows results for each trait. 

.Figures 6-10 are included to show the progressive nature of average 
pre and post sum scores for each course type by grade level. In every 
instance, average sum scores move progressively higher a* the course type 
becomes more demanding. Since raters had no idea which course type 
they were scoring during the scoring process, these findings appear to 
indicate that students in differentiated course types do indeed possess 
different levels of sk<ll in writing. Furthermore, students in more 
demanding course types possess higher levels of skill in writing than those 
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in less demanding course types. Based on these data, one could conclude 
that students arc appropriately assigned to English course levels. 
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Table 6 

Comparison of Pre and Post Paragraph Scores for Compensatory Education Classes (10,11) 







dumber 


of Students who Experienced: 








Negative (-) 
Change from 
Pre to Post 


wo Change (0) 
from 
Pre to Post 


| Positive (+) 
Change from 
Pre to Post 


Level 


Grade /Trait 


N 


of 

Significance 


Grade 10 


52 








• 


'Composing 
Sentence 
Formation 
Usage 
Mechanics 




14 

14 
14 
16 


12 

16 
18 
18 


26 

22 
18 
18 


.01* 

.16 { 

.39 

.56 


♦SUM 




19 


4 


29 


.05* 


Grade 11 


39 










Composing 
Sentence 

Formation 
Usage 
♦Mechanics 
SUM 




16 

12 
12 
8 
16 


14 

12 

16 . 
10 
6 


■ 9 

15 
11 
21 
17 


.28 

.78 
.58 
.07* 
.69 



•denotes statistical significance p£.10 
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Figure I. Mean Scores for: 
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Table 7 

Comparison of Pre and Post Paragraph Scores for Basic Classes (10,11,12) 





Number of Students Who Experienced: 




Grade/Trait 


N 


Negative (-) 
Change from 
Pre to Post 


No Change (0) 
from 
Pre to Post 


Positive (+) 
Change from 
Pre to Post 


Level 
of 

Significance 


Grade 10 


96 










♦Composing 
"bentence 




25 


31 


40 


.00* 












Formation 




26 


24 


46 


.00* 


Usage 




28 


30 


38 




^Mechanics 




20 


34 


42 


.01* 


**SUM 




35 


H 


■j i 


.00* 


Grade 1 1 


138 


















Composing 
♦Sentence 




45 


33 


60 


.19 












Formation 




31 


53 


54 


.00* 


*Usage 




32 


50 


S6 


.01* 


♦Mechanics 




32 


48 


58 


.00* 


*SUM 




50 


11 


77 0 


.01* 


Grade 12 


49 










Composing 




15 


14 


20 


.48 


Sentence 












Formation 




16 


14 


19 


.37 


♦Usage 




11 


18 


20 


.06* 


Mechanics 




11 


20 


18 


.16 


SUM 




18 


6 


25 


.15 



♦denotes statistical significance p.*10 
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Figure 2. Mean Scores for: 
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Table 8 

Compan ion of Pre and Post Paragraph /Essay Scores for Regular Classes (10,11,12) 





Number 


of Studtnts who Experienced: 




Grade / Trait 


N 


—Negative (-) 
Change from 
Pre to Post 


No Change (0) 
from 
Pre to Post 


Positive ( + ) 
Change from 
Pre to Post 


Level 
of 

Significance 


Grade 10 


100 










Composing 
Sentence 

Formation 
Usage 
Mechanics 




.42 

45 
32 
29 


22 

24 
36 
35 


36 

31 
32 
36 


' .97 

.26 
.71 
.64 


SUM 




48 


10 


42 


.64 


irl -» r-J ■ 1 4 

urade i 1 


89 










Composing 
♦Sentence 

Formation 
Usage 
Mechanics 




31 

25 
22 
24 


24 

26 
34 
40 


. 34 

38 
33 
25 


.55 

.03* 

.12 

.56 


*SUM 




31 


9 


49 


.08* 


Grade 12 


106 










♦Composing 
Sentence 

Formation 
Usage 
Mechanics 




32 

35 
32 
22 


25 

28 
40 
46 


49 

43 
34 
38 


.03* 

.26 
.93 
.23 


♦SUM 




39 


14 


53 


.08* 



*denotes stat ; stical significance of p£.10 
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Table 9 



Comparison of Pre and Post Essay Scores for Advanced Classes (10,11,12) 



Grade /Trait 



Grade 10 

Composing 
Sentence 
Formation 
Usage 
Mechanics 

SUM 



Grade 11 

♦Composing 
♦Sentence 

Formation 
♦Usage 
•Mechanics 

*SUM 



N 



82 



108 



Number of Students who Experienced: 
Negative (-) I No Change (0) 



Change from 
Pre to Post 



28 

30 
23 
21 

34 



28 

18 
26 
24 

27 



No Change 
from 
Pre to Post 



30 

24 
30 
30 



24 

37 
38 
37 

15 



Positive (+) 
Change from 
Pre to Post 



24 

28 
29 
31 

41 



56 

53 
44 
47 

66 



Level 
of 

Significance 



.62 

.93 
27 
.29 

.42 



.00* 

.00* 
.05* 
.01* 

.00* 



Grade 12 

Composing 
•Sentence 
Formation 
Usage 
Mechanics 

SUM 



96 



38 

26 
28 
19 

32 



20 

30 
39 
41 

17 



38 

40 
29 
36 

47 



.96 

.10* 

.94 

.19 

.32 



♦denotes statistical significance p^.10 
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Table 10 

Comparison of Pre and Post Essay Scores for AP Classes (10,11,12) 



Grade/Trait 



Grade 10 

*Com posing 
Sentence 

Formation 
Usage 
Mechanics 

♦SUM 



Grade 1 I 

Composing 
♦Sentence 

Formation 
Usage 
Mechanics 

SUM 



Grade 12 

Composing 
Sentence 
Formation 
Usage 
Mechanics 

SUM 



N 



99 



95 



95 



Number of Students who Experienced 



Negative (-) 
Chanqe from 
Pre to Post 



24 

25 
31 
30 

34 



30 

23 
27 
31 

33 



33 

36 
30 
35 

41 



•denotes statistical significance p<.10 



"WChange (GJ 
from 
Pre to Post 



27 

33 
36 
31 

13 



23 

34 
36 
34 

13 



19 

29 
34 
30 

13 



Positive ( + ) 
Change from 
Pre to Post 



48 

41 
32 
38 

52 



42 

38 
32 
30 

49 



43 

30 
31 
30 

41 



Level 
of 

Significance 



.00* 

.60 
.45 

.01* 



,11 

,04* 

,40 
,95 

25 



46 



.26 
.84 
.60 

.98 
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Figure 5, 
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Table 11 
Summary Table 
Traits in Which Significant Improvement Occurred 



Grade 


Compensatory 
Education 


Basic 


Regular 


Advanced 


AP 


Total 


10 


Composing 
Overall 


Sentence 

Formation 
Mechanics 
Overall 


None 


None 


Composing 

Overall 


- 8/25 areas' 
improved 
(32%) 


11 


Mechanics 


Sentence 

Formation 
Usage 
Mechanics 
Overall 


Sentence 

Formation 
Overall 


Composing 
Sentence 

Formation 
Usage 
Mechanics 
Overall 


Sentence 
Formation 


13/25 areas 
improved 
(52%) 


12 


Not Assessed 


Usage 


Composing 
Overall 


Sentence 
Formation 


None 


4/20 areas 
improved 
(20%) 


Total 


3/10 areas 
improved 
30% 


9/15 areas 
improved 
60% 


4/15 areas 
improved 
27% 


6/15 areas 
improved 
40% 


3/15 areas 
improved 
20% 
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Table 12 



Summary Table -- Trait Improvement 





Number of Course Levels 
in which improvement occurred 


Percent of 
Total 


Composing 


• 

5 


36% 


Sentence Formation 


6 


43% 


Usage 


3 


21% 


Mechanics 


4 


28% 


Overal 1 


7 


50% 
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Figure 7 
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Figure 8 



ALL TRAITS — SUM SCORE 



\ 
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POST 



PRE 
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Figure 9 



ALL TRAITS — SUM SCORE 



16.0, 
15.0 

m 

14.0, 
13.0_ 
12.0, 
11.0, 
10. 0, 
9.0 



(10.05) 



(11.52) 




.(10.39) 



(12.90) 



8.0 



i r 



PRE POST 
— REGULAR- - 



i i i i r 

PRE POST 
- A0VANCED - 



l 1 1 1 1 

PRE POST 
AP 



GRADE LEVEL: ELEVEN (ESSAY) 
COURSES: REGULAR, ADVANCED , AP 



50 

43 



■t 



16.0. 
15. 0_ 
14.0 



Figure 10 
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5.0 CONCLUSIONS/RECOMMENDATIONS 



5.1 Conclusions 

Based on product data from the 1983-84 program evaluation' of 
the Writing Enhancement Program for tenth , eleventh, and twelfth 
grade students in Hillsborough County, the following conclusions can 
be drawn. 

1) In general, the program was effective in improving 
students 1 writing. Students in 78 percent (11 of 14)' 
of senior high course levels improved some aspect of 
their writing skills. 

2) By course level: Basic students, especially those in 
grades 10 and 11, showed the most improvement 
overall . 

3) By grade level: Eleventh grade students, especially 
those in basic classes and advanced classes showed 
improvement in the most areas. These two course 
levels also showed the most improvement during 
1982-83; in that year eleventh grade advanced stu- 
dents showed the most gain for essays and eleventh 
grade basic students showed the most gain for para- 
graphs. 

4) Students in three co. rse levels, tenth grade regular, 
tenth grade advanced, and twelfth grade AP, showed 
no improvement in any assessed area. Based on. their 
pre-scores , twelfth grade AP students may already 
have possessed reasonable skills in all assessed areas 
and therefore not have shown significant growth 
during the school year. 



\ 

5) Most student growth occurred in two areas, composing 
and/or sentence formation. Whenever students signif- 
icantly improved in the composing trait, their overall 
writing ability also improved significantly. 

Recommendations 

The following recommendations are based on conclusions listed in 
section 5.1 and take into account results of the previous year's 
evaluation . 

1) Maintain present curriculum delivery techniques and 
activities in the following course levels: tenth grade 
basic, eleventh grade basic, and eleventh grade ad- 
vanced. They appear to be highly effective. 

2) Investigate delivery techniques and activities in the 
following course levels: tenth grade regular and tenth 
grade advanced. They appear to have lit'tle or no 
impact on student writing skills. 

3) Determine extent to which instruction in the composing 
area affects overall writing performance . A highly- 
positive relationship may exist between these two 
occurrences, and instruction in this domain may en- 
hance student writing abilities to a greater extent than 
instruction in other areas* 

4) Writing samples written by students during 1983-84, 
the procedures and materials used in the scoring 
process, and evaluative findings should be used to 
provide inservice for senior high school teachers 
involved in the Writing Enhancement Program, Such 
insv^rvice would have implications both for writing 
instruction and for evaluation of student work in the 
classroom . 
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5) As soon as possible > findings of this report should be 
disseminated to all senior high school teachers involved 
in the Writing Enhancement Program. Discussion of 
district-level and school-site program strengths and 
weaknesses should emanate from these findings. 

6) Writing Enhancement Program evaluation for the 
1984-85 school year should emphasize tenth and elev- 
enth grade course levels and follow the same scoring 
procedures as 1983-84. New prompts (topics) should 
be generated and used to elicit the pre and post 
student writing samples. As suggested by several 
Writing Enhancement Program teachers,, prompts should 
be designed to elicit writing in the third person, 
Pre-writing samples should be written the *ast week in 
September; post samples should be written the first 
week in May. 
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Paragraph Topics and Procedures 
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MEMORANDUM 



Selected Writing Enhancement Teachers 
Compensatory Education, Basic, Regular 

Alvany Wilson, Supervisor, Secondary English 
Lore A. Nielsen, Coordinator, Testing & Evaluation 

September 21, 1983 

Product Evaluation of Writing Enhancement 
Program - Paragraphs 



Enclosed is an overview of the program evaluation of the senior high 
school Writing Enhancement project. You have been selected to parti- 
cipate in our data collection efforts. Please follow the procedures 
below to collect a writing sample during the week of October 3-7. 



The day before students are scheduled to write, remind them to 
bring a pen to class the next day. 

On the writing day, tell the students the following: 

"Today you are going to write a paragraph. The reason I 
you are writing now is so that at the end of the year 
we can compare your work with another sample of your 
writing to see if you have improved your skills." 

Distribute the assignment sheets and blonk vjrms. Read the 
assignment sheets alouci while students read them silently. 

If students ask how long their writing should be, tell them 
to try to keep 1t to one page. !f their writing must be 
longer, have them complete It on a sheet of notebook paper 
with their name in the upper right-hand corner. Staple the 
second page to the first. 
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TO: 

FROM: 
DATE: 
SUBJECT-. 



-Page I 



Give no Individual help during the writing time. 

After ten minutes tell students that 1f they are still planning, 
they should beglfl writing now. 

After 30 minutes, tell students the/ have five minutes to bring 
their papers to a close. 

Tell students that if they finish early, they are not to leave 
their seats. Direct them to read a book or to some other seat 
activity. 

Collect all materials. Check all writing samples to make sure 
there 1s a legible firsthand last name on eacn . Alphabetize 
each set of writing samples I 

Try to collect writing samples from all students, including those who 
are absent on the day the class writes. However, do not collect any 
samples after October 12. 

Also, please include an alphabetical class roster along with the w rit- 
ing samples . Indicate m which class period students are writing. 

Because this 1s an evaluation, please do not discuss the topics with 
your students or use them for any other purpose. 

Return all materials no later than October 14 to Lore Nielsen, Depart- 
ment of Testing and Evaluation, School Administrative Center. If you 
have any questions or concerns, feel free to contact Lore Nielsen at 
272-4341. 

Thank you for your cooperation. 

AW:LAN:dsr 
Enclosures 
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Hillsborough County Public Schools 
Writing Enhancement Program - Paragraph Sample 

SET I 



ST UDENT DIRECTIONS 

1. Fill 1n the top of the blank page. 

2. Choose one of the two topics listed below. Write a paragraph of 

at least five sentences on that topic. 

3. Write with a pen. 

4. Indent the first line and write 1n proper paragraph form. 

5. Write or print legibly. 

6. Do not use a dictionary or ask your teacher for help. 

7. Do not give your pape** a title. 

8. You have 35 minutes to do your writing. You should not write a 
rough draft. However, you should plan what you want w say before 
you begin writing. You may use scratch paper for your planning. 

9. Read over your writing when you finish to make any corrections. 
. 10. Correct errors by crossing out neatly. 



SET I — TOPICS 

A. Think about the person you admire most. Give the reasons you have 
for choosing that person. 

B. Some people believe a woman's place is 1n the home. Others do not. 
Which do you believe? Give your reasons. 
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Hillsborough County Public Schools 
Writing Enhancement Program - Paragraph Sample 

SET II 



STUDENT DIRECTIONS 

1. Fill 1n the top of the blank page. 

2. Choose one of the two topics listed below. Write a paragraph of - 

at least five sentences on that topic. : 1 

3. Write with a pen. 

X 

4. Indent the first Une and write 1n proper paragraph form. 

5. Write or print legibly. 

6. Do not use a dictionary or ask your teacher for help. 

7. Do not give your paper a title. 

8. You havs 35 minutes to do your writing. You should not write a 
rough draft. However, you should plan what you want to say before 
you begin writing. You may use scratch paper for your planning. 

9. Read over your writing when you finish to make any corrections. 
10. Correct errors by crossing out neatly. 



SET II— TOPICS 



A. When you have free time, what do you. like to do the most? Describe 
your favorite sport, hobby, or other pastime and explain why you en- 
joy 1t. 

B. What would you change about your school 1f you could change one thing? 
Give your reasons for wanting that change. 
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Hillsborou gh County Public Schools 
. writing Enndncernent Program 



PARAGRAPH SAMPLE 



Name 
School 



Date 



Teacher/Period 



Grade (circle one) 10 11 12 



Course level (circle one) C.E. 



BASIC 



• < 



\ 



Appendix B 
Essay Topics .and Procedures 
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Joa E. Nawaoma. Vice Chairman 

Roland H. Lewta 

A. Laon Lowry 

R. Sonn? Palomino 

Sam Rampailo 

Marion S. Rodgars 

SUPERINTENDENT OF SCHOOLS 
Raymond O Shalton 

SCHOOL ADMINISTRATIVE CENT* 
901 East Kannady Blvd (Tampa) 
(813) 277 4O00 
SUNCOM 571-4000 



MEMORANDUM 



TO: 

FROM: 
DATE: 
SUBJECT: 



Selected Writing Enhancement Teachers 
Regular, Advanced, AP 

Alv?ny Wilson, Supervisor, Secondary English 

Lore A. Nielsen, Coordinator, Testing & Evaluation 

September 21, 1983 

Product Evaluation of Writing 
Enhancement Program - Essays 



Enclosed 1s an overview of the program evaluation of the senior high 
school Writing Enhancement project. You have been selected to parti- 
cipate in our data collection efforts. Please follow the procedures 
below to collect a writing sample during the week of October 3-7. 




The day before students are scheduled to wri\.e, remind them to 
bring a pen to class the next day. 

On the writing day, tell the students the following: \ 

"Today you are going to write an essay. The reason you ^ 
are writing now is so that at the end of the year we 
can compare your work with another sample of your writ- 
ing to see if you have improved your skills." 

Distribute the assignment sheets and blank forms. Read the 
assignment sheets aloud while students read them silently. 

If students ask how long their writing should be, te'll them 
it should be no longer than two pages. If their writing must 
be longer, have them complete it on a sheet of notebook paper 
with their name in the upper-right-ha id corner. Staple the 
page to the rest of the essay. 
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Page 2 



Give no Individual help during the writing time. 

After ten minutes tell students that 1f they are stW planning, 
they should begin writing now. 

After 40 minutes, tell students they have five minutes to bring 
their papers to a close. 

Tell students that 1f they finish early, they are not to leave 
their seats. Direct them to read a book or to some other seat 
activity. 

Collect all materials. Check all writing samples to make sure 
there is a legible first and last name on each . Alphabetize 
each set of writing samples" 



Try to collect writing samples from all students, Including those 
who are absent on the day the class writes. However, do not collect 
any samples after October 12. 

Also, include an alphabetical class roster with the writing samples . 
I nd 1 cate m which class period students are writing. 

Because this 1s an evaluation, please do not discuss the topics with 
your students or use them for any other purpose. 

Return all materials no later than October 14 to Lore Nielsen, Depart- 
ment of Testing and Evaluation, School Administrative Center. If you 
have any questions or concerns, feel free to contact Lore Nielsen at 
272-4341. 

Thank you for your cooperation. 

AW:LAN:dsr 
Enclosure 



Hillsborough County Public Schools 
Writing Enhancement Program - Essay Sample 




STUDENT DIRECTIONS 

1. Complete the required Information at the top of the page, 

2. Write with a pen. 

3. Use proper essay form. 

4. Write or print legibly. 

5. Correct errors by crossing out neatly. 

6. Do not use a dictionary or ask your teacher for help. 

7. Do not give your paper a title. 

8. You have 45 minutes to do your writing. You should not write 
a rough draft. However , you should plan what you want to say 
before you begin writing. You may use scratch paper for your 
planning. 

9. Read over your writing when you finish to make any corrections. 



TOPIC - A 

If you could make an event 1n American history happen again so that you 
could play a part in it, which one would you choose? Write an essay in 
which you describe the event and explain why you chose that particular 
event and what part you would like to pla.y in it. 



I 



Hillsborough County PufrH c Schools 
Writing Enhancement Program - 'Essay Sample 



STUDENT DIRECTIONS 

X. Complete the required Information at the top of the page. 

2. Write with a pen. 

3. Use proper essay form. 

4. Write or print legibly. 

5. Correct errors by crossing out neatly. 

6. Do not use a dictionary or ask your teacher for help. 

7. Do not give your paper a title. 

8. You have 45 minutes to do your writing. You should not write 
a rough draft. However, you should plan what you want"to say 
before you begin writing. You may use scratch paper for your 
planning. 

9. Read over your writing when you finish to make any corrections 



TOPIC - B 

Most of us look up to some famous person as a representative of the things we 
believe 1n or as the kind of person we would like to be. Think about a famous 
person whom you admire. Select a particularly admirable characteristic or 
quality of that person. Write an essay describing this characteristic or quality. 
Be sure to provide an illustration of it from the person's life. Try to show that 
the person is great at least partly because of this characteristic or quality. 
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Hillsborough County Public Schools 
Writing Enhancement Program - Essay Sample 

Name Da te 

School Teacher/ Period 

Grade (circle one) 10 11 12 Course level (circle one) Regular Advanced / 



Appendix C 

Writing Enhancement Program Scoring Logistics 
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SCORING, LOGISTICS 
HILLSBOROUGH COUNTY SCORING PROJECT 
WRITING ENHANCEMENT PROGRAM 



This document details logistical procedures to be followed in paper prepa- 
ration, training, scoring, and follow-up activities. The schedule for completing 
all activities in Section 1.0 must be followed. If you anticipate any delays, con- 
tact Kitty immediately so that arrangements to assist you can made. 



1.0. PAPER PREPARATION 

^Although paper p reparatio n for Pa rap:raphj_must be independen t from 
^prep aration for Es says, Jhe proced ures ^will be identical. To prevent con- 
fusion, however,. Section 1.1 describes Paragraph preparation, and Sec- 
tion 1.2 restates procedures as £hey apply to Essay preparation. 



1. Organize all pre-test papers in 7 sets, as follows: 



PgC - Grade 10 
PgC - Grade 11 
PgC - Grade 12 
PgB - Grade 10 
PgB - Grade 11 
PgB - Grade 12 
PgR - Grade in 



("Pg" is used to designate "Paragraph"; " n " , " B " , and "R" are 
used to designate Compensatory, Basic, . -1 Regular levels, 
respectively.) 

1. Prepare a list of the names of students in each set. Alphabetize 
each list to break up clusters by class, and school. 

3. Organize all post-test papers in 7 sets identical lo those named 
in #1, above. 

4. Compare the post-test papers in each set to the appropriate pre- 
test list. Use a check to designate students whose post-test 

*- paper is present. When you have finished each set, remove all 
post-test papers for students whose names did not appear on 
your pre-test list. Then, remove all pre-test napers for stu- 
dents whose names are not checked on your lirt, and draw <i line 
through the unchecked names on your list. When this step is 
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completed , each pre-test set and its corresponding post-test set 
will constitute completely matched pairs. 

5. Count the number of students iemaining on the list(s) for each 
of the levels: Compensatory, Basic, and Regular. 

6. Determine the proportion of each level represented by each grade 
level. Use the following example as a guide. 

Number of Grade in students in PgC (n of PgC-10) = 80n 

Number of Grade 11 students in PgC (n of PgC-11) = 400 

Number of Grade 12 students in PgC (n of PgC-12) = 200 

Total Number of PgC students (N of PgC) = 1400 

The Proportion (P) of Grade X students in PgC is equal to -- +z T 

Therefore: » 

P PgC-. 0 = .571 

P PgC-11 = .286 

P PgC-12 = .143 



7. Using the total sample size for each level, determine the number 
of students from each grade level to be included in the sample. 
To do this, you will multiply the total sample size for the level 
by the proportion for the grade level 7" Use the example below 
as a ^uide. 

The sample size for PgC (SPgC) is 300 




(P Grade X) (SPgC) = n of SPgC-X 
Therefore: 

(.571) ( ^300) p 171 Grade in Students 
(.286) (300) = 86 Grade 11 Students 
(. 143) (300) = 43 Grade 12 Students 

8. To determine the sampling interval for each level, divide the 
^-total leval's N by the total level's sample size. Using the PgC 

example above, you would divide 1400 by 300 to get a sampling 
interval of 5 (with rounding), 

9. Once the sampling interval has been determined, you are ready 
to identify the scoring sample by marking every nth student on 

FR?r eacn <C rade list for the level. For example, using the Pp-C data 



from the example above, you would identify everv 5th student on 
the grade 10 list, every 5th student on the grade 11 list, and 
every 5th student on the grade 12 list. To begin, select a 
™?. ent .arbitrarily a nd at random from anywhere on the grade 
1U list. (If you feel that you cannot be arbitrary and random, 
please use a Table of Random Numbers to identify the first stu- 
dent.) Mark this name with a colored pen. Then count down 
five names from this one, skipping over all crossed out names. 
Mark the fifth name. Continue marking every fifth name until 
you have marked 171 names. You should have gone through 
your entire list and started over. Use this procedure to identifv 
students on each of your 7 lists. 

10. Pull the pre-test paper for each identified student from the de- 
signated set, for all sevWsets. When you have finished this 
fite P. y ou will have s even new sets of paDers. 

11. You now need\o pull additional papers from the original £ sets 
for pre-scoring.\ To do so, please randomly identify six more 
students on each of your lists. Try "to distribute your selection 
across the entire list. Pull the pre-test paper for each student. 
You should, have pulled one paper for each of 42 students. Set 
tnes^ pre- scoring papers aside. 



12. Now you will assign identification numbers to each of the stu- 
dents in the scoring sample. First, put together all of your 
grade lists for each level (e.g., lists for trades 10, 11, and 12 
for group PgC). Begin with the first identified student on the 
grade 10 list and assign the number 001; assign the next iden- 

, JJ !! udent the number 002. Continue until you have assigned 
a 3-digit number to each identified student on your grade 10 list 
for the group. DO NOT ASSIGN I.D. NUMBERS TO STUDENTS 
IN THE PRE-SCORING SAMPLE! In our PgC example, you would 
have numbers 001 through 171 on your grade 10 list. Assign the 
next counting number to the first student on your grade 11 list 
for the group. In our example, the first grade 11 student would 
be 172; the last one woula be 257. Continue with your tirade 12 
list in the same manner. In our example, grade 12 would have 
students 258 through 300. 

** Each level (PgC, PgB, and PgR) will begin with 001. ** 

13. You are now ready to code the pre-test papers, A 7-d-igit numer- 
ical code will be recorded on each paper, as follows: 

^ — Positions 1^2, a 3 w ill be used for the student i.d. number 

Positioi/4> ill indicate occasion: 

Pre-test = 1 
Post-test - 2 
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— Position(V)will indicate^ roup : 

PgC = 1 
PgB = 2 ^ 
^^PgR^= 3 ^ ^ ' 

— Position 6 will represent the topic 

Set I - Topic A = 1 
Set I - Topic B = 2 
Set II - Topic A = 3 
^-^Sfttll - Topic B = 4 



f: J. 



Position Jl__ will represent the student's g radej ^ eve^ and will 
be separated from the first 6 digits by Vdash." 

Record the codes at the rif?ht, just below the course level informa- 
tion on the Student Writing Sample. See illustration below. 




HllUborough County Public School* 
! Writing Enhancement Prog raw 



PARA6RAPH SAMPLE 



Net* ^P-iUy J)4*sk<^s 



Grade (circle one) (to) 11 12 



Teacher/Period 



Date s/zt>J*2 



Course level (circle "fane) CCTCQ BASIC Rli. 

dot 3.) 3 - to 



5 



14. As you code the papers, place them in ascending order by i.d, 
number within level. When you finish, you will have three 
stacks, one each for PgC, PgB, and PgR. In the PptC and PgB 
stacks you wili have one paper each for approximately 300 stu- 
dents. In the PgR stack, you will have one paper for each of 
approximately BO students. 

Repeat steps 9 through 14 for all post-test sets. When you 
record codes on these papers, BE SURE that you use the correct 
i.d. number for each student. The i.d. for an individual stu- 
dent is the same for both writing samples, so, altogether, eacn 
number will be assigned to two writing samples, one pre-test and 
two post- test, in each level. ^ 



Kitty will pick up the 84 papers selected for pre-scorine (#11, 
above). She will return them to you to copy and assemble for 
pre-scoring. After pre-scoring, Kitty will give you instructions 
about prepari ng the pape rs for the sc oring session. 

17. You now have 6 stacks of papers, one pre-test and one post-test 
stack for each of levels PgC, PgB, and PgR. 

18. You will organize these six stacks into 22 stacks of approximately 
64 papers, each. Use the following procedures to ensure that 
group clusters are interrupted. 

"Deal" the top paper in each of the six stacks, one 
to each of 6 "hands." Rotating in the same manner, 
continue dealing one paper at a time until you have 
established 22 hands. When you have 1 paper in 
each of 22 hands, continue dealing in rotation with 
the second paper for each hand. Proceed with this 
dealing pattern until you have dealt all papers. When 
you have finished, you should have 22 stacks of ap- 
proximately 64 papers each. 

19. You must now brtak each of these stacks into fi scoring sets, as 
follows : 

Starting at the top of the stack, count in papers and 
clip them togr/.'^i' with a large paper clip. Continue 
through the entire **ack. Combine your last full set 
of 10 papers with the remaining papers in the stack. 
You should have 5 sets of 10 papers each, plus one set 
of approximately 13 papers. Follow this procedure with 
each of the 22 stacks. 

Pick up the 5 10-paper sets from each stack and out these sets 
"-"together. Then pick up the 13-paper sets and put them to- 
gether. 



15. 



16. 
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20. Now you will remove 'all identifying information from papers and 
assemble them in scoring packets. Begin with one of the full 
sets and proceed as follows: 

Using a paper-cutter, carefully slice away the top of 
each writing sample page just below the line with 
grade and course level information. When, you have 
sliced all papers in a scoring set , staple the papers 
together securely. Proceed in the same manner with 
each set until you have sliced and assembled 132 sets. 

21. Now you will prepare a scoring document for each paper, as 
follows : 

a. Locate the first 6-dififits in the code on the first paper 
in the set. Record this number on the scoring docu- 
ment in the top left corner. You will record the num- 
ber vertically. (See illustration below.) 

b. Locate the parade level (last number, following dash) 
and record it in the box at the bottom of the "Instruc- 
tions" section. (See illustration, below.) 




7-1 



c:st copy > 



V 



22. 



23: 



\ 



24. 
25. 



Once the codes have been transferred to the scoring document, 
you should darken the bubble that corresponds to each number, 
as shown above. 

When a scoring document has been prepared for each paper in 
the set, place the scoring documents in the same order as the 
student papers. Then place the oapers and the scoring docu- 
ments in a clear plastic bag (to be provided by PDE). 

Repeat steps 21-23 for each set of papers. 

Once all paper sets and corresponding scoring documents have 
been assembled in plastic bags, each bag should be sealed with 
a Control Label (to be provided by PDE). Number each set, 
beginning with 10-paper sets. Record the set number on the 
Control Label, as shown. You should have a total of 132 sets. 

CONTROL LABEL 
WEP - PARAGRAPH 
Set No: 01 



1st Rdr: 
2nd Rdr: 



NNA: 



26. 



Place all packets in boxes for transfer to the scoring site. Use 
a colored slip sheet to separate 10-paper sets from 13-paper sets 



1.2 Paper Preparation A Essay 

■ 1. Organize all pre-test papers in 8 sets, as follows: 



ER - 


Grade 


11 


ER - 


Grade 


12 


EA - 


Grade 


10 


EA - 


Grade 


11 


EA - 


Grade 


12 


EAP - 


Grade 


10 


EA P - 


Grade 


11 


EAP - 


Grade 


12 
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,("E" is used to designate "Essay"; ff R", n A f \ and "AP" are used 
to designate Regular, Advanced and AP levels, respectively.) 

Prepare a list of the names of students in each set. AlDhabetize 
each list to break up clusters by „ss, and school. 

i 

Organize all post-test papers in 8 sets identical to those named 
in #1, above. 



8 



4. Compare the post- test papers in each set to the appropriate pre- 
test list. Use a check to designate students whose post-test 
paper is present. When ycu have finished each set, remove all 
pojst-test papers for students whose names did not appear on 
your pre-test list. Then, remove, all pre-test papers for stu- 
dents whose names are;, not checked on- your list; and draw a line 
through the^ unchecked names' on your list. ~ V/hen this step is 
completed , , each pre-test set and its 1 corresponding post-test- set 
will constitute completely matched pairs. . 

5. Count the number of students remaining on the list(s) for fe^ch 
of the levels* Regular, Advanced and AP. 



6. Determine the proportion of each level represented by each gra^'.c 
level. Use the following example as a j^uide. 

Number of Grade 10 students in EA (n of EA-10) v= S00 

Number, of Grade 11 students in EA (n of EA-11) = 4no 

Number of Grade 12 students in EA (n of EA-12) = 200 

Total Number of EA students (N of EA) - 14nn 

*" n -X 
The Proportion (P) of Grade X students in EA is equal to r — 

Therefore: ' ) * 

P EA-10 = .571 
P EA-11 = .286 
P EA-12 = .143 

7. Using the total sample size for each level, determine the number 
of students from each grade level to be included in the sample. 
To do this, you will multiply the total sample size for the. level 
by the proportion for the grade level. Use the example below 
as a guiuj. t — 

The sample size for EA (SEA) is 300 

(P Grade X) (SEA) = n of SEA-X ' 
Therefore : 

\ 
\ 

(.571) (300) = 171 Grade 10 Students 
(.288) (300) = 86 Grade 11 Students 
(.143) (300) = 43 Grade 12 Students 
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8. 



9. 




10. 



11. 



T-) determine the sampling interval for each level, divide the 
total level's N by the total level's sample size. Using the EA 
example above, you would divide 1400 by 300 to get a sampling 
interval of 5 (with rounding! 

Once the sampling interval has been determined, you are ready 
to identify the scoring sample by marking every nth student on 
each grade list for the level. For example, usine the EA data 
from th example above, you would identify every 5th student on 
the grade 10 list, every 5th student on the grade 11 list, and 
every 5th student on the grade 12 list. To begin, select a 
student arbitrarily and at random from anywhere on the grade 
10 list. (If you feel that you cannot be arbitrary and random, 
please use a Table of Random Numbers to identify the first stu- 
dent.) Mp*k this name with a colored pen. Then count down 
five names "from this one, skipping over all crossed out names, 
Mark the fifth name. Continue marking every fifth name until 
you have marked 171 names. You should have gone through 
your entire list and started over. Use this procedure to identify 
students on each of your 8 lists. 

Pull the pre-test paper for each identified student from the de- 
signated set, for all eight sets. When you have finished this 
step, you will have^ight new sets of papers 



You now need to pull additional papers from the original 8 sets 
for pre-scoring. To do so, please randomly identify six more 
students on each of your lists. Try to distribute your selection 
across the entire list. Pull the pre-test paper for each student. 
You should have pulled one paper for each of 48 students. Set 
these pre-scoring papers aside. 
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12. Now you will assign identification numbers to each of the stu- 
dents in the scoring sample. First, put together all of your 
grade lists for each level (e.g., lists for grades in, n, and 12 
for group EA). Begin with the first identified student on the 
grade 10 list and assign the number 001; assign the next iden- 
tified student the number 002. Continue until you have assigned 
a 3-digit number to each identified student on your grade 10 list 
for the group. DO NOT ASSIGN I.D. NUMBERS TO STUDENTS 
IN THE PRE-SCORING SAMPLE! In our EA example, you would 
have numbers 001 through 171 on your grade 10 list. Assign the 
next counting number to the first student on your grade 11 list 
for. the group. In our example, the first grade 11 student would 
be 172; the last one would be 257. Continue with your grade 12 
list in the same manner. In our example, grade 12 would have 
students 258 through 300. 

^»» Each level (ER, EA, and EAP) will begin with OfU. ** 

13. You are now ready to code the pre-test papers. A 7-digit numer- 
ical code will be recorded on each paper, as follows: 

77 



in 



— Positions^l, 2, s 3 w ill be used for the student i.d. number 

— ^Qaition-4-will indicate occasion: 

Pre-test = 1 
Post-test = 2 




-y Position 6 Mil represent the topic : 
Topic A ■ 1 
Toole B = 2 




— ,Po^ition___7_^Hl represent the student's grade level and will 
be separated from the first 6 digits by a dash. 

Record the codes at the right, just below the course level infor- 
mation on the Student Writing Sample. See illustration on page 4. 

14. As you code the papers, place them in ascending order by i.d. 
number within level. When you finish, you will have three 
stacks, one each for ER, EA, and EAP. In the ER stack, you 
will have one paper each for approximately 200 students. In the 
EA and EAP stacks, you will have one paper for each of approxi- 
mately 300 students. 

15. Repeat steps 9 through 14 for all post-test sets. When you 
record codes on these papers, BE SURE that you use the correct 
i.d. number for each student. The i.d. for an individual stu- 
dent is the same for both writing samples, so, altogether, each 
number will be assigned to two writing samples, one pre-test and 
two - post- test , in each level. 

16. Kitty will pick up the 96 paDers selected for pre-scoring (#11, 
above). She will return them to you to copy and assemble for 
pre-scoring. After pre-scoring, Kitty will *dve you instructions 
about preparing the papers for the scoring session. 

17. You now have 6 stacks of papers, one pre-test and one post-test 
^tack for each of levels ER, EA , and EAP 

18. You will organize these six stacks into 22 stacks of approximately 
73 papers, each. Use the following procedures to ensure that 
group clusters are interrupted. 
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"Deal" the top paper in each of the six stacks, one 
to each of 6 "hands." Rotating in the same manner, 
continue dealing one paper at a time until you have 
established 22 hands. When you have 1 paper in 
each of 22 hands, continue dealing" in rotation with 
the second paper for each hand. Proceed with this 
dealing pattern until you have dealt all papers. When 
you have finished , you should have 22 stacks of ap- 
proximately 73 papers each. 

You must now break each of these stacks into 7 scoring sets, as 
follows: 

Starting at the top of the stack, count in papers and 
clip them together with a large paper clip. Continue 
through the entire stack. Combine your last full set 
of 10 papers with the remaining papers in the stack. 
You should have 6 sets of 10 papers each, plus one set 
of approximately 13 papers. Follow this procedure with 
each of the 22 stacks. 

Pick up the 6 10-paper sets from each stack and put these sets 
together. Then pick up the 13-paper sets and put them to- 
gether. 

Now you will remove all identifying information from papers and 
assemble them in scoring packets. Begin with one of the full 
sets and proceed as follows: 

• 

Using a paper-cutter, carefully slice away the top of 
each writing sample page just be low the line with 
grade and course level information When you have 
sliced all papers in a scoring set, staple the papers 
together securely. Proceed in the same manner with 
each set until you have sliced and assembled 154 sets. 

Now you will prepare a scoring document for each paper, as 
follows: 

a. Locate the first 6-dicrits in the code on the first paper 
in the set. Record this number on the scoring docu- 
ment in the top left corner. You will record the num- 
ber vertically. (See illustration on page 6.) 

b. Locate the grade level (last number, following dash) 
and record it in the box at the bottom of the "Instruc- 
tions" section. (See illustration on pae:e 6.) 

Once the codes have been transferred to the scoring document, 
you should darken the bubble that corresponds to each number, 
as shown above. 
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23. When a scoring document has been prepared for each paper in 
the set, place the scoring documents in the same order as the 
student papers. Then place the papers and the scoring docu- 
ments in a clea- plastic bag (to be provided by PDE). 

24. Repeat steps 21-23 for each set of paoers. 

25. Once all paper sets and corresponding scoring documents ht*ve 
been assembled in plastic bags, each bag should be sealed with 
a Control Label (to be provided by PDE). Number each set, 
beginning with 10-paper sets. Record the set number on the 
Control Label, as shown. You should have a total of 154 sets. 

CONTROL LABEL 
WEP - ESSAY 
Set No: 01 

1st Rdr: NNA: 

2nd Rdr: 



26. Place all packets in boxes for transfer to the scoring site. Use 
a colored slip sheet to separate 10-paper sets from 13-paper 
sets. 

ADDITIONAL MATERIALS 

1. Prepare name tags as follows: 

a. One white tag for each of twenty-two Readers: 

Reader No: 

Name: 



Although there will be 22 readers, they will not be assigned Reader 
Numbers in a consecutive sequence. Readers' Assigned Numbers are 
given below. 



Reader 


Assigned 1 


1st 


01 


2nd 


02 


3rd 


03 


4th 


04 


5th 


10 


6th 


11 


7th 


12 


8th 


13 


9 th 


14 


10th 


20 


11th 


21 


12th 


22 


13th 


23 


14th 


24 


15th 


30 



SO 
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Reade r Assigned Number 

16th 31 

17th 32 

18th 33 

19th 34 

20th 40 

21st , 41 

22nd 42 



b. One pinktag for each of three assistants: 

SCORING ASSISTANT 
Name: 



c. One green tag for the Trainer: 



TRAINER 

/ 

Name : 



2. The following additional materials will be needed at the scoring site. 

f a. 10 Extra Name Tegs (Yellow) 
\ b. 75 No. 2 lead pencils 
) c. Several red pens 
< d. 2 staplers and a box of staples 
e* 2 scissors 

f. 6 rolls of H" x 2*" WRITE GUARD (250/roll) 

g. 50 - 75 extra scoring documents 

3.0 TRAINING 

Training and scoring for the Paragraph will be conducted first, and 
will be followed by training and scoring for the Essay. In both cases, 
procedures will be the same. After preliminary activities are finished, the 
Trainer will conduct a 15-20 minute discussion of the Scoring procedures 
and will review the Manual for another 10 minutes. Then, a total of five 
Training Rounds will be conducted. Your 'activities will be the same for 
each round. First, you will give each Reader the Training packet desig- 
nated by the Trainer, and collect the packets as Readers finish scoring. 
When you have collected the packets, proceed as follows: 

a. Open each packet and remove the scoring documents. Record 
the Reader's domain scores under the respective True Scores 
for each paper on the Monitor Sheet for the round. Then, com- 
c-"pare the Reader's score with the True Score for each domain, 
for each paper, and record the difference by recording: 

(1) A "0" for each score that is the same as the True Score. 

(2) A if the Reader's score is higher than the True Score, 
and the number of points higher the Reader's score is. 

ERIC § 1 
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(3) A "-" if the Reader's score is lower than the True Score, 
and the number of points lower the Reader's score is. 

b. Compute the Difference for each domain and record the total in 
the appropriate column under "Tendency." It is important that 
you remember to record the correct sign. 

c. Total the number of non-zero Difference values, and record this 
number under the appropriate column labeled "Incidence." 

10-a, b, and c are illustrated below. 



MONITOR SHEET 
DOMAIN A 
TRAINING ROUND # 


Paper No. 


01 


02 


03 04 05 


Tendency 


Incidence 


True Score 


4 


2 


1 2 3 






Reader 


3 


2 


2 4 4 






Difference 


-1 


0 


+1 +2 +1 


+3 


4 


Reader 












Difference 













4.0 SCORING 

1. The scoring room should be set up as diagrammed below 
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When formal scoring begins you will give each Reader one scoring' 
packet. 

As each Reader finishes scoring the first packet, you will pick up the 
scored packet, checking to be sure that the Reader has recorded 
his/her i.d . number on the Control Label , and rrive the Reader 
another packet. 

As Readers ar<i scoring the second packets, you will open the scored 
packets and remove the scoring documents. Check 'each document to 
be sure that: 

a. the Reader has writ ten and coded i n his/her i.d* number. 

b. the Reader has lecorded a score for each domains. 

c. the Reader has written and coded in the same number for 
each Score . 

d. the Reader has completely and correctly coded in scores. 

Use the following rules to alter scoring documents: 

a* If the Reader has either written or coded in his/her i.d., 
but failed to do both, you should complete the entry. 

b. If the Reader has failed to record his/her i.d., you should 
check the Control Label and complete the entry. 

c. If the Reader has either written or coded all scores but 
has failed to do both, you should complete the entry, 

d. If the Reader has written in one score and coded in a 
different score, and /or if the Reader has failed to enter all 
required scores, you should MAKF NO CHANGE. Record 
the paper i.d. number, the Reader No., and the Packet 
No. on a Scoring Irregularities Form. Comolete the Form 
and give it and the packet to the Trainer. 

e. If the Reader has not completely coded in or erased data., 
you should do so, using a #2 lead pencil to darken codes. 

When each document has been checked, place a WRITE GUARD over all 
scores. Check to be sure that scoring documents are in the same 
order as the Writing Samples, and put the scoring documents hac&Mx 
the plastic bag. 

Place the packet in front of the Second Read «r , The Se 2onu Read er 
will- be the reade~ sitting to the immediate left of the First Reader. 
If the First Reader is 01, the Second Reader will be 0*2; if the First 
Reader is 02, the Seeorr* Reader will be 03, and so on. If the Fir: .t 
Reader is 42 t the Second Reader will be 01. 



The procedure described in steps 3-6 will be followed throughout 
scoring. no 
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BEST COPY AVAILABLF 



When a packet has been scored by its 2nd Reader, collect the packet 
and follow the procedures in #4, above, to check the documents. 
Then process the packet, as follows: 



a. Peel off the blinding layer of WRITE GUARD from each document. 

b. Each Reader will have assigned four scores, one for each of four 
domains. Compare the scores assigned for each domain on each 
cocument. Set aside all documents on which any two domain 
scores are different by more than one point (i.e., 1 and 3, 
1 and 4, 2 and 4). 



c. Count the number of score pairs on which any two domain scores 
differ by more than one score. Record the number of discrepant 

score pairs on the Control Label, beside "NNA ". Remembe-* 

that the number you record for NNA should reflect the number 
of discrepant score pairs, not the number of documents. 

d. Place the scoring documents in "c M on top of the Writing Samples, 
and place all other scorinpr documents on the bottom, under fie 
Writing Samples. Then place the entire stack back in iU plastic 
bag. 

e. Stack the bags from "d" separately from other scoring bags. 

The Trainer will check these completed Control Labels to assess 
the agreement level. If the Trainer determines that rescoring of dis- 
crepant papers is necessary, you will prepare for rescoring as follows: 

a. Identify all packets on which the Control Label shows 

a number of 1 or greater beside "NNA: ". (These 

are packets from 8-e, above.) Circle the FTNA number 
in red ink. 

b. C^en the scoring packet and remove the scoring docu- 
ments on t. 7. Check the documents to be sure that at 
*east one domain's scores are different by more than one 
number. Count the discrepant score pairs to be sure that 
the number of pairs corresponds to the number circled in 
red on the Control Labe 1 . 



c. On each of these documents, using a RED PErf ONLY, place 
a small check beJde the domain or domains for which there 
are discrepant score pairs. (See illustration below .) 



Jim. 
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d. Place a WRITE GUARD over eac* of the two sets of scores. 

e. Place these documents on too of the Writing Samples and 
return all materials to the scoring packet. 

f. Stack packets for rescoring. 

10. An accuracy check will be conducted periodically. The Trainer will 
tell you when it is time for a Check Round. These Rounds usually 
will follow a coffee break or a lunch break. When it is time for a 
Check Round, you will give each Reader the Check Round packet 
designated by the Trainer, and collect the packets as Readers finish 
scoring When you have collected the packets, oroceed exactly as you 
did during the Training rounds, using check Round packets and ap- 
p: opriately labeled Monitor Sheets. 

FOLLOW-UP ACTIVITIES 

1. Open the scoring packet and remove all answer documents. Leave 
the writing samples in the bag, and reseal the bag. 

2. Theck each document to be sure that: 

a. the following information is both written and coded in: 

(1) Student i.d. number 

(2) Student grade level 

(3) 1st Reader's i.d. number 

(4) 1st Reader's scores 

(5) 2nd Reader's i,d. number 

(6) 2nd Reader's scores 

*(7) 3rd Reader's i.d. number 
3rd Reader's scores 

b. All coding is neat and complete. 

c. All erasures are clean. 

3. Stack °J1 scoring documents in the same direction. 

4. V/hen ail packet have been opened and processed, stack all scoring 
documents neatly and transfer them to data processing. 



Appendix D 
Writing Enhancement Program List of Raters 
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List of Raters 



Maggie Allen 
Bertha Baker 
Patricia Bishop 
Elaine Blanco 
Jean Bradley 
Stephanie Collore 
Ann Cook 
Deborah Damon 
Susan DiFederico 
Stan Lelm 
Kathy Mabry 
Edwina Mayer 
Yolanda Menendez 
Patricia Parker 
Valerie Lynn Mercak 
Joseph Pennarrhic 
Jeanette Robinson 
Brenda Ross 
Deborah Shep?** i 
Tenripress fv ion 
Catherine Steinker 
Diane Woodbury 



Brandon Senior High School 
Leto Comprehensive High School 
East Bay Senior High School 
Gaither Senior High School 
Tampa Bay Vo-Tech High School 
Robinson Senior High School 
Hillsboroi'.gh Senior High School 
East Bay Senior High School 
Robinson Senior High School 
King Senior High School 
Leto Comprehensive Hif h , School 
King Senior HigV. School 
King Senior High School 
Hillsborough Senior High School 
Brandon Senior h'gh School 
Jefferson Senior High School 
Brandon Senior High School 
Hillsborough Senior High School 
Leto Comprehensive High School 
Chamberlain Senior High School 
Robinson Senior High School 
Chamberlain Senior Hi^h School 
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INTRODUCTION 



Florida's Writing Enhancement Program (WEP) has just completed its third 
year. During the first two years of the program, all hierh school juniors and 
seniors in Hillsborough County were served. In its third year, the legislature 
mandated that all students enrolled in senior high schools must be served by 
the program. In response to this mandate, during the 1983-84 school year, the 
Hillsborough County WEP program was expanded to serve all sophomores en- 
rolled in senior high schools. 

The evaluation of the 1982-83 Writing Enhancement Program showed that 
the program was being implemented according to guidelines and that it was 
viewed by most participants as being successful. In addition, results indicated 



that students in four of the six evaluated course levels showed significant im- 



provement in paragraph and essay writing. 



The purpose of the 1983-84 evaluation is to determine i: the expanded 
program is being implemented according to guidelines and if student writing 
improves as a result of participation in the program. This scoring activity is 
part of a larger evaluation effort to make these determinations. Results of the 
scoring activity will be used to provide answers to two important research 
questions: 

j Are paragraphs or essays written by high school students at the end of a 
s'' year's instruction better than paragraphs or essays written by the same 
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i 



students at the beginning of the year? ^ 



i 



In what specific ways does the writing of Writing Enhancement students 
improve after involvement in the writing program? J 



To provide as much information as possible in answering this question, the 
Department 6t Testing and Evaluation has designed a com orehen dive evaluation 
which includes examining change among multiple grade and instructional levels. 



This Manual will be used to conduct all scoring for the Writing Enhancement 
Program. Paragraphs and Essays will be scored separately. Chapter II will 
guide Paragraph scoring, and chapter III will fruide Essay scoring. 

Your participation in this scoring activity is critically important to the 
integrity of the evaluation results, and. we appreciate your enthusiastic involve- 
ment. You will receive extensive training for handscoring writing samples, and 
we hope that the training will continue to be valuable to you long after our 
formal scoring activities are over. 




ISSUES IN HANDSCORING WRITING SAMPLES 



Most hand-scoring procedures currently in use throughout the country can 
be classified under two rubrics : hoMstic and analytical , Holistic scoring re- 
quires that the Reader evaluate the overall quality of a piece by permitting all 
characteristics of the piece to interact with each other in influencing his/her 
judgment. Analytical scoring requires that the Reader isolate each particular 
characteristic under examination and judge each without being: influenced by 
any of the others ♦ 

Decisions about which of these two approaches should be used in a scoring 
effort must be made in light of the type of information that one needs or de- 
sires about a program. The information that is needed about this program — 
whether students write better after a year in the program, and in what ways 
their writing improves — requires a judgment of quality regarding specific char- 
acteristics of the composition. For that reason, a special scoring procedure of 
t>he holistic type has been chosen for use. 

Pure holistic scoring has certain measurement properties which cause it to 
be inappropriate for this evaluation, however. Briefly, because pure holistic 
scoring is normative in nature, it does not provide the kind of information 
about performance levels that this evaluation effort requires. Furthermore, 
pure holistic scoring does not provide for observation of specific characterisitcs. 
To acquire the necessary information while maintaining an assessment of overall 
quality, a variation of holistic scoring, called focused holistic domain scoring:, 
will be used. Focused holistic scoring is the most widely adopu-d of all hand- 
scoring models currently used in lartre-scale scorirg programs. Its popularity 
can be attributed to its easy adaptability to program purposes and to the in- 
structional value inherent in the results that it yields. Adapting focused 
holistic scoring to independent evaluation of composition domains provides highly 
discrete information about the quality of stuc ent writing. 

When hand-scoring activities are used to n^ake significant decisions about 
examinees or to provide data in a formal research study, certain controls must 
be imposed. Many of the procedural controls imposed during preparation and 



'during actual scoring are designed with two objectives in mind: that the scores 
be velid and that they be reliable. Score validity reflects how accurate the 
scores are. Score reliability reflects how consistent the Readers are. 

A score is considered valid if it accurately reflects the examinee's actual 
performance, or the examinee's "true score." In a performance testing program 
such as a writing- assessment, "true score" is determined by expert judgment. 
Score validity is monitored throughout scoring by periodically comparing Reader 
scores with the experts' "true scores," assigned during pre-scoring. Before 
the scoring session convenes, a group of experts is assembled to conduct 
pre-scoring activities. During pre-scoring ,• these experts, who represent the 
school system's present standards, undergo intensive training. After they have 
been trained, they score a large number of papers. This scoring is followed by 
extensive discussion and re-scoring, until the expert group is in agreement 
about the score assigned to each paper. Some~of these pre-scored papers 
become Anchor Papers, so called because they anchor or stabalize the scale. 
The Anchor Papers are included in the Scoring Manual and are used to train 
Readers. Other pre-scored papers are used to monitor Reader accuracy at 
periodic intervals throughout the scoring session. These are called Checl- 
Papers because they are used to check whether Readers are continuing to score 
in concert with the designated criteria or are beginning to drift away from 
these criteria. 

When it is lime to conduct an accuracy check, each Reader is given copies 
of the same set of papers. When Readers have scored these papers, each ' 
Reader's scores are compared with the scores assigned by the experts. Each 
Reader is given an Accuracy Rating, and these ratings are returned to the 
Reader, along with each paper's expert or "true scores" and an explanation for 
the "true scores." Readers whose Accuracy Rating falls within an acceptable 
range are asked simply to review the "true scores" and the rationale state- 
ments. Readers whose Accuracy Ratings are not within the range of tolerance 
are required to^ undergo further training before continuing in formal scoring 
activities. *~~ 

A test is considered to be reliable if it consistently yields the same inc ''ca- 
tion of examinee achievement, knowledge, or performance. Several different 
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procedures often are used to evaluate how reliable a test is. Score reliability 
is evaluated by examining a property closely related to internal consistency and 
it is attained when two or more observations of performance agree with or are 
the came as each other. Costs associated with hand-scoring often prohibit our 
judging examinee performance on the basis of more than one writing sample, 
however. In the case of hand-scoring, therefore, we define observations not 
as items, but as ratings, each Reader-assigned score constituting an observa- 
tion . 

To ensure that ratings are reliable, another type of moni oring is con- 
ducted throughout scoring activities. This monitoring is not periodic, as in the 
case of the accuracy checks. Rather, it is on-going. Each paoer is scored by 
two or more different Readers. To guarantee that each set of scores is inde- 
pendent, the first set score is obscured from the view of subsequent Readers. 
Once a set of papers has been scored by all of its Readers, the scores on each 
paper are compared. The more often all of the scores are the same, the more 
reliable the scores are judged to be. .Tust as some deviation can be tolerated in 
the Accuracy Rating, some deviation also can be tolerated in the Agreement 
Rating. If a set of Readers begins to stray too far from perfect agreement, 
however, their most recent Accuracy Ratings are checked to determine whether 
one or more of the Readers :s inaccurately applying the score scale. The inac- 
curate Reader or Readers is then removed from formal scoring and retrained 
until they have regained sufficient Accuracy to be reinstated. 

Large-scale scoring activities such as this one require enormous planning 
and organization efforts both before and during the formal scoring session, not 
only to ensure that scores are accurate and reliable, but to be sure that the 
paper flow is smooth, that all scores are independent, and that scoring is 
reasonably distributed across all Readers. F^om time to rime durincr the scoring 
session. Readers are asked to wait quietly while the scoring staff completes 
calculations necessary for monitoring accuracy and consistency . Patient and 
cooperative Realers always are appreciated. 
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SCORING PROCEDURES AND REGULATIONS 



Because each of you has a background that includes some study of and /or 
experience in writing, all of you share some knowledge and understanding in 
common • The training and experience that you bring to this scoring activity 
will, for the most part, serve you well. They will enable you to read and 
score compositions with an informed eye. 

Just as each of you brings special knowledge and experience tc this ac- 
tivity, however, each also brings special biases and prejudices about the re- 
lative importance and appropriateness of certain characteristics of written com- 
position. These prejudces and peeves can vary significantly from one Reader 
to another. 

The "jst objective of training is to capitalize on the common knowledge 
and understanding among Readers, and to remove, at least temporarily , the 
biases and special peeves that they might hold. The training material is de- 
signed to give clear definition to each characteristic that you will be asked to 
evaluate, in every instance where possible, the Scoring Guide reduces judge- 
ment and inference to the lowest possible level. Even so, judgement and infer- 
ence are necessary. It will be essential, therefore, that you accept the defini- 
tions set forth in this Guide, notwithstanding your own feeling that you might 
assign a different score if you could use your own standards to rate the compo- 
sition. The tension between your own standards and those imposed in this 
Guide will frustrate you from time to time. It mi^ht be of some comfort to you 
to know that everyone who ever was involved in a scoring activity like this one 
has experienced all of the frustrations that you are about to experience and 
that all of them have survived. 

Each composition will be scored by two Readers. Our expectation is that 
the score assigned by each Reader will be the same in almost all cases. Be- 
cause each at" you * r in^core^ it 
is important that all of you reach a common understanding of the definitions and 
of how to apply them. Your consistent agreement with other Readers and your 
accurate adherence to the scoring criteria will be monitored throughout the 
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scoring session. If it becomes apparent that your scoring has become inconsis- 
tent or inaccurate, you will be required to undergo retraining exercises. 

The Trainer will answer your procedural questions and will clarify scoring 
criteria, if necessary. The Trainer will not, however, advise you about the 
score that a composition should receive. 

Several rules must be observed during the scoring session: 

1. Each Reader's score must be the result of an independent evaluation 
of the composition. You will not be allowed to confer with another 
Reader regarding a composition. If you are tempted to share some- 
thing particularly delightful with a fellow Reader, suppress the urge 
and read on. 

2. It is important that a quiet atmosphere be maintained at all times. If 
you need clarification or assistance, raise your hand and the Trainer 
will come to you. In conferring with the Trainer, speak as quietly as 
possible so that other Readers will not be distracted by your conver- 
sation . 

3. Scoring efficiency is important. You will need to concentrate on each 
composition so that one reading will allow you to assign all scores 
without re-reading or re-scanning the composition. Reasonable speed 
is important, so concentrate and establish an efficient pace. 

4. If you need to take a break to stretch your legs, you may quietly 
walk around in the designated area of the scoring room, or you may 
leave the scoring room for a few minutes. Two 15-minute breaks will 
be provided each day, one in the morning and one in the afternoon. 
In addition, you will be given a one-hour lunch break. You will be 
expected to accomplish 6} hours of actual scoring each day. 

5. Prolonged near-point focus can cause eye fatigue. If you begin to 
experience tiredness or burning in your eyes, take a few minutes to 
look up and focus your eyes on a distant point. This will help to 
prevent headaches and blurred vision. 



6. Under no circumstance should you remove scorinp materials from the 
scoring room. When you take a break, you should leave all of your 
materials on the table in front of you. 

Your after-hours activities are not governed by this project. Nonetheless, 
you are encou raped to rest your eyes and pet a pood nipht's sleep each nipht. 
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SCOFUWG THE PARAGRAPH 
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EXPECTATIONS FOR THE COMPOSITION 

« 

Altogether , four— tQ£ic§^ were used as prompts for eliciting a paragraph. 
Two of the topics, I-A and II-A, are mildly personal. The central composing 
effect in paragraphs written for Topic I-A is expected to be d-scriptive, al- 
though an explanatory explication strategy may be used. The central com- 
posing effect for paragraphs written for Topic I-B is expected to be explana- 
tory. 

Both topics I-B and II-B are somewhat argumentative in nature, although 
I-B more clearly sets up argumentative writing than does II-B. In response to 
either of these topics, the writer is expected to communicate an opinion, then 
employ an explanatory explication strategy to develop it. Some writers may 
adopt a purpose beyond simple explanation, attempting to persuade readers to 
embrace th opinion about a woman's place, or attempting to convince readers 
that the proposed school change is needed. 

The same criteria will be used to score all paragraphs, regardless of the 
topic. All of the characteristics to be evaluated have been grouped under four 
domains: composing, senten ce formation, usage, an d mechanics . A discussion 
of each domain is provided below. 

COMPOSING: The Composing Domain comprises all of the characteristics of a . 

composition that influence the effective delivery of the writer's 
message to achieve the writer's purpose. These characteristics 
include specifying and focusing on the subject-matter, pioviding 
information that explicates the subject-matter, and delivering the 
explication through organized, logical, related, and coherent 
discourse. The traits specified for observation in this domain 
are: 

Topic Sentence : A topic sentence is a sentence that 
summarizes or generalizes the main idea of the 
paragraph. It does not necessarily have to be 
the first sentence. 

100 
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Supports ; Supports are reesons or examples which fit 
the topic logically and which are relevant. 

Transitions ; Transitional words and phrases link 
expressions, information, and ideas. 

^ Conclusion ; A concluding statement finishes the com- 

| position and gives it a quality of completeness. 

It may be a restatement of the main idea, a sum- 
mary, or some other device which completes the 
treatment of the subject. 

SENTENCE FORMATION: The Sentence Formation domain comprises all of the 
characteristics of composition that influence the effective expres- 
sion of assertions, propositions, questions, commands, and 
exclamations such that T-units and their constituents are inter- 
nally related and are either separated from or coordinated with 
each other. These characteristics include variety in sentence 
length and structure, and the absence of formation errors such 
as run-ons, fragments, misplaced modifiers, faulty pronoun 
references, dangling modifiers and nonparalled elements. The 
traits specified for observation in this domain are; 

Sentence Construction : Sentence Construction addres- 
ses the formation of complete sentences and the 
absence of structural errors resulting in run-ons, 
fragments, misplaced modifiers, faulty pronoun 
references, dangling modifiers, and non-parallel 
elements. A complete sentence is a crroup of 
words having a subject and predicate, expressing 
a complete thought, and marked by terminal punc- 
*~ tuation. A run-on is a group of words marked as 

a sentence by capitalization and /or terminal punc- 
tuation, but whi, . contains two or more indepen- 
dent clauses not joined by appropriate punctuation 
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or by a coordinating conjunction. A comma is 
considered appropriate between two independent 
clauses only if the two are "contact clauses" and 
neither contains more than eight words. 

Sentence Variety ; Sentence variety is achieved when 
the writer uses a combination of the four^ sentence 
structures: simple, compound, comDlex» and 
compound-complex. A simple sentence has one 
Independent clause and no subordinate clauses. 
A compound sentence has two or more indepen- 
dent clauses but has no subordinate clauses. A 
complex sentence has one independent clause and 
one or more subordinate clauses. A compound- 
complex sentence has at least two independent 
clauses and one or more subordinate clauses. 

The Usage Domain comprises all of the characteristics of lan- 
guage selection and use that cause a composition to be acceptable 
and effective for standard, informal, discourse intended for an 
audience committed to national, reputable, and present stan- 
dards. These characteristics include the selection of language 
appropriate for the purpose of the piece, the presence of 
agreement in number between subject noun and verb elements, 
the agreement in number, gender and case between nouns and 
pronouns, and the use of correct and appropriate inflected 
forms. The traits to be observed in this domain are: 

Subject-predicate usage : Correct usage includes tense 
formation and agreement in number and person. 

* • » 
Pronoun usage : Correct usage includes pronoun- 

antecedent agreement in number and /or gender, 

and appropriateness of nominative or objective 

case. 
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Precise verbs ; Precise verbs are verbs that elicit 
specific images in the reader's mind. 

Descriptive modifiers ; Descriptive modifiers are ad- 
jectives or adverbs that elicit specific images in 
the reader's mind. 

The Mechanics Domain is the only one of the four domains which 
is restricted exclusively to written language. This domain 
comprises 'the system of symbols and cuing devices that allow us 
to represent language in written text. The symbol system 
includes punctuation, capitalization, and the graphemic symbols 
that we commonly refer to as spelling. The translatability, or 
legibility of these symbols also is classified in the Mechanics 
Domain. Cuing devices such as paragraph indentation and text 
formatting also are members of the Mechanics Domain. The traits 
to be observed in this domain are: 

Capitalization : All capitalization rules will be under 
examination, including those related to unneces- 
sary capitalization. 

Commas : All rules governing the use of commas for 
items in a series, date*, appositives, direct ad- 
dress, introductory phrases, and before conjunc- 
tions linking independent clauses of more than 
eight words will be under examination. The use 
of unnecessary commas also will be evaluated. 

Apostrophes : All rules governing the use of apostro- 
phes in contractions and possesslves, and the 
ommission of apostroDhes in possessive oronouns 
will be examined. The use of unnecessary apos- 
trophes also will be evaluated. 
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Paragraph Format : The composition must have only 
one paragraph, which must be inc. nted at the 
beginning* 

Legiblity : Legibility will be judged on clear formation 
of letters, spacing of words and sentences, and 
neatness of corrections. The paragraph can be 
printed or written in cursive. 

Spelling : The spelling of all words used in the para- 
graph will be evaluated. 
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FOCUSED HOLISTIC 
SCORE SCALE 

COMPOSING DOMAIN 

The paragraph reveals the writer's skilled command of composing skills . 
The paragraph has a clear topic sentence. The writer has developed the 
topic by providing supports which clearly focus on and explicate the 
subject-matter. The writer has achieved logic and cohesiveness by using 
transitional elements to establish clear relationshipr between and among 
units of information and to move the discourse forward. The paragraph 
has a quality of completeness. 

The paragraph reveals the writer's reasonable composing control . The com- 
position has a topic sentence. The writer has developed the topic by pro- 
viding supports which adequately focus on and explicate the subject-matter, 
although minor retreats may be present. In spite of possible minor gaps, 
the writer has achieved adequate logic and cohesiveness by using sufficient 
transitional elements to establish relationships between and among units of 
information and to move the discourse forward. The paragraph has ade- 
quate closure. 

The paragraph reveals the writer's minimal composing control . The writer 
may have provided a topic sentence, but development is minimal and focus 
may suffer from frequent retreats. Although the writer may have provided 
information about the subject-matter, his/her minimal use of transitional 
elements may cause the paragraph logic and cohesiveness to be marginal. 
Although the writer may have written a summary or restatement, closure 
may be incomplete. 

The paragraph reveals little if any * evidence of the writer's composing 
control . The writer's subject-matter may be marginally discernible, but it 
is undeveloped. The writer may have provided some information about 
his/her subject-matter, but relationships between and among units of infor- 
mation ftray be so weak as to seriously mar the paragraph's logic and cohe- 
siveness. The writer may have failed to establish reasonable closure. 

The paragraph annot be scored . YOU MUST HAVE THE TRAINER'S 
AUTHORIZATION TO ASSIGN THIS SCORE. 
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FOCUSED HOLISTIC 
SCORE SCALE 



SENTENCE, FORMATION DOMAIN 



The paragraph reveals the writer's skilled command of sentence form ation 
skills. The paragraph is free of structural misformations and contains a 
variety of sentence types. 

The paragraph reveals th e writer's reasonable control of sentence forma- 
tion skills. Structural misformations are few and minor, and the writer 
has used a reasonable variety of sentence types. 

The paragraph reveals the writer's minimal control of sentence formation 
skills. Structural misformations cause the reader to cuestion the writer's 
skill. The writer may have attempted to vary sentence types, but skills 
may be marginal. 

The paragrap h reveals little if any evidence of the writer's control of sen- 
tence formation skills. Structural misformations predominate. The writer's 
control of variety in sentence type is not discernible. 

The paragr aph cannot be scored . YOU MUST HAVE THE TRAINER'S 
AUTHORIZATION TO ASSIGN THIS SCORE. 
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FOCUSED HOLISTIC 
SCORE SCALE 

USAGE DOMAIN 

4 = The paragraph reveal s the writer's skilled consistent command of Standard 
American English usagt: . The paragraph is free of non-standard usage and 
of usage errors. The writer has selected language which enhances the 
composition's effectiveness. 

3 = The paragraph reveals the writer's reasonable and generally consistent 
control of Standard American English usage . The writer may have com- 
mitted minor usage errors. The writer's language is reasonable and does 

> 

not detract from the paragraph effectiveness. 

2 = The paragraph reveals the writer's minimal and inconsistent control of 
Standard American English Usage . Non-standard usage may cause the 
reader to question the writer's understanding of standard forms. The 
writer's language may detract from the paragraph's effectiveness. 

1 = The paragraph reveals little if any evidence of the writer's control of 
Standard American English usage . The paragraph may be seriously marred 
by the writer's use of non-standard forms; The writer's language may 
seriously impede the paragraph's effectiveness. 

0 = The paragraph cannot be scored . YOU MUST HAVE THE TRAINER'S 
AUTHORIZATION TO ASSIGN THIS SCORE. \ 

\ 

\ 
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FOCUSED HOLISTIC 
SCORE SCALE 

MECHANICS DOMAIN 

The paragraph rev^s the writer's skilled and consistent command of the 
m echanics of written discourse . The paragraph is free of errors in capi- 
talization, comma and apostrophe use, and spelling. The paragraph is 
indented and the handwriting is clearly legible. 

The parag r aph reveals the writer's reasonable and generally consistent 
control of the mechanics of written dis c curse . Although errors in capital- 
ization, comma and apostrophe use, or spelling may appear, they are 
minor. The writer's intent to use paragraph format is clear, and his/her 
handwriting is reasonable* 

The paragraph reveals the writer's minimal and inconsistent control of 
the mechanics of written discourse . Errors in capitalization, comma and 
apostrophe use, and spelling are serious. The writer's intent to format a 
paragraph may not be clear. The handwriting may be difficult to read. 

The paragraph reveals little if any evidence of the writer's control 
of the mechanics of written discourse . Errors in capitalization , comma and 
apostrophe use, and spelling are obtrusive. The writer's intent to format 
a paragraph may not be clear. The handwriting may be marginally legible. 

The paragraph cannot be scored . YOU MUST HAVE THE TRAINER'S 
AUTHORIZATION TO ASSIGN T£IS SCORE . 
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EXPECTATIONS FOR THE COMPOSITION 

Both prompts are expected to elicit mildly personal essays. The central 
composing effect for essays written in response to PromDt A is expected to be 
explanatory, although writers probably will use a narrative-descriptive explica- 
tion strategy. The central composing effect for Prompt B is expected to be 
descriptive. Writers may, however, utilize an explanatory explication strategy. 

The same criteria will be used to score essays written for both topics. All 
of the characteristics to be evaluated have been grouped under four domains: 
composing, sentence formation, usage, and mechanics. A discussion of each 
domain is provided below. 



COMPOSING: The Composing Domain comprises all of the characteristics of a 

composition that influence the effective delivery of the writer's 
message to achieve the writer's purpose. These characteristics 
include specifying and focusing on the subject-matter, providing 
information that, explicates the subject-matter, and delivering the 
explication through organized, logical, related, and ccherent 
discourse. The traits specified for observation in this domain 
are: 

Thesis Statement : A thesis statement is defined as a 
sentence that summarizes or generalizes the main 
idea of the essay. It should be positioned near 
the beginning of the essay. 
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Supports : Supports are reasons or examples which fit 
the thesis statement logically and which are rele- 
vant. 

Transitions ; Transitional words and Dhrases link 
expressions, information, and ideas. 

Dev elopment / Or ganiza tion : Development and organiza- 
tion are defined as the stated logical progression 
of ideas from thesis statement to conclusion* 

Conclusion : A conclusion is defined as a sentence(s) 
or paragraph which finishes the paper and gives 
the reader a sense of completion. It may be a 
restatement of the thesis, a summary or some 
other means to complete the treatment of the sub- 
ject. 



SENTENCE FORMATION: The Sentence Formation domain c^.nprises all of the 

characteristics of composition that influence the effective expres- 
sion of assertions, propositions, questions, commands, and 
exclamations such that T-units and their constituents are inter- 
nally related and are either separated from or coordinated . ith 
each other. These characteristics include variety in sentence 
length and structure, and the absence of formation errors such 
as run-ons, fragments, misplaced modifiers, faulty pronoun 
references, dangling modifiers and nonparalled elements. The 
traits specified for observation in this domain are: 

Sentence Construction : Sentence Construction addres- 
ses the formation of complete sentences and the 
absence of structural errors resulting in run-ons, 
fragments, misplaced modifiers, faulty pronoun 
references, dangling modifiers, and non-Darallel 
elements. A complete sentence is a group of 
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words bavin* a subject and predicate, expressing 
a complete though* , and marked by terminal punc- 
tuation. A run-on is a group of words marked as 
a sentence by capitalization and/or terminal punc- 
tuation, but which contains two or more indepen- 
dent clauses not joined by appropriate punctuation 
or by a coordinating conjunction. A comma is 
considered appropriate between two independent 
clauses only if the two are "contact clauses" and 
neither contains more than eight words. 

Sentence Variety: Sentence variety is achieved when 
the writer uses a combination of the four sentence 
* structures: simple, compound, complex, and 
compound-complex. A simple sentence has one 
independent clause and no subordinate clauses. 
A compound sentence has two or more indepen- 
dent clauses but has no subordinate clauses. A 
complex sentence has one independent clause and 
one or more subordinate clauses. A compound - 
complex sentence has at least two 'independent 
clauses and one or more subordinate clauses. 

USAGE: The Usage Domain comprises all of the characteristics of lan- 

guage selection and use that cause a composition to be acceDtable 
and effective for standard, informal, discourse intended for an 
audience committed to national, reputable, and present stan- 
dards. These characteristics include the selection of language 
appropriate for the purpose of the piece, the presence of 
agreement in number between subject noun and verb elements, 
the agreement in number, sender and case between nouns and 
pronouns, and the use of correct and appropriate inflected 
«-fcrms. The traits to be observed in this domain are: 

Subject-pre dicate usage s Correct usage includes tense 
formation and aorreement in number and person. 
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Pronoun usage ; Correct usaee includes pronoun- 
antecedent agreement in number and/or gender, 
and appropriateness of nominative or objective 
case. 

Precise verbs : Precise verbs are verbs that elicit 
specific images in the reader's mind. 

Descriptive modifiers : Descriptive modifiers are ad- 
jectives or adverbs that elicit specific images in 
the reader's mind. 

MECHANICS: The Mechanics Domain is the only one of the four domains which 
is restricted exclusively to written language. This domain 
comprises the system of symbols and cuing devices that allow us 
to represent language in written text. The symbol system 
includes punctuation, capitalization, and the graphemic symbols 
that we commonly refer to as spelling. The translatability , or 
legibility of these symbols also is classified in the Mechanics 
Domain. Cuing devices such 'as paragraph indentation and text 
formatting also are members of the Mechanics Domain. The traits 
to be observed in this domain are: 

Capitalization : All capitalization rules will be under 
examination, including those related to unneces- 
sary capitalization. 

Commas : All rules governing the use of commas for 
items in a series, dates, appositives, d'.rect ad- 
dress, introductory phrases, and before conjunc- 
tions linking independent clauses of more than 
0 eight words will be under examination. The use 

of unnecessary commas also will be evaluated. 
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Apostrophes : All rules governinc; the use of apostro- 
phes in contractions and possessives, and the 
ommission of apostrophes in possessive pronouns 
will be examined. The use of unnecessary apos- 
trophes also will be evaluated. 

Paragraph Format : The composition must have only 
one paragraph, which must be indented at the 
beginning. 

Legiblity : Legibility will be jud fired on clear formation 
of letters, spacing of words and sentences, and 
neatness of corrections. The paragraph can be 
printed or written in cursive. 

Spelling : The spelling of all words used in the para- 
graph will be evaluated. 
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FOCUSED HOLISTIC 
SCORE SCALE 

COMPOSING PdMAIN 

The essay reveals the writer's skilled command of composing skills . The 
essay has a clear thesis statement. The writer has developed the these 
by providing supports which clearly focus on and explicate the subject- 
matter. The essay is well organized and logical. The writer has achieved 
coheshreness by using transitional elements to establish clear relationships 
between and among units of information and to move the discourse forward. 
The essay's conclusion is strong and effective. 

The essay reveals the writer's reasonable composing control . The essay 
has a thesis statement. The writer has developed the thesis by providing 
supports which adequately focus on and explicate the subject-matter, al- 
though minor retreats may be present.. The essay is reasonably organized 
and logical. In spite of possible minor gaps, the writer has achieved 
adequate coheshreness by using sufficient transitional elements to establish 
relationships between and among units of information and to move the dis- 
course forward. The writer establishes an adequate sense of closure. 

The essay reveals the writer's minimal composing control . The writer may 
have provided a thesis statement, but development is minimal and focus 
may suffer from frequent retreats. Although the writer may have provided 
information about the subject-matter, the essay may suffer from disruptive 
gaps in organization and logic. Minimal use of transitional elements may 
cause the essay's coheshreness to be marginal. Although the writer lay 
have written a summary or restatement, closure may be incomplete. 

The essay reveals little if any evidence of the writer's composing control . 
The thesis may be marginally discernible, but it is undeveloped. The 
writer may have provided some information about the subject-matter, but 
relationships between and among units of information may be so weak as to 
seriouslyMnar the essay's organization, logic, and coheshreness. The writer 
may have failed to establish reasonable closure. 

The essay cannot be scored . YOU MUST HAVE THE TRAINER'S AUTHOR- 
IZATION TO ASSIGN THIS SCORE. 
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FOCUSED HOLISTIC 
SCORE SCALE 



SENTENCE FORMATION DOMAIN 



4 s The essay reveals the writer's skilled oommand of sentence formation skills . 
The essay is free of structural informations and contains a variety of sen- 
tence types. 

3 s The essay reveals the writer's reasonnhte conlrgLof sentence formation 
skills. Structural misformations are few and minor, and the writer has 
used a reasonable variety of sentence types, 

2 = The essay reveals the w riter's minimal control of sentence formation skills . 
Structural misformations cause the reader to question the writer's skill. 
The writer may have attempted to vary sentence types, but skills may be 
marginal . 



1 = The essay r eveals little if any evidence of the writer's control of sen- 
tence formation skills. Structural misformations predominate. The writer's 
control of variety in sentence tyoe is not discernible. 

0 = T he essay cannot be scored . YOU MUST HAVE THE TRAINER'S AUTHOR- 
IZATION TO ASSIGN THIS SCORE. 
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FOCUSED HOLISTIC 
SCORE SCALE 



USAGE DOMAIN , 

The essay reveals the writer's skilled consistent command of Standard 
American English usage . The essay is free ok non-standard usage and of 
usage errors. The writer has selected language which enhances the compo- 
sition's effectiveness. ^ 

The essay reveals the writer's reasonable and generally consistent control 
of Standard American English usage . The writer may have committed minor 
usage errors. The writer's language is reasonable and does not detract 
from the essay's effectiveness. 

The essay reveals the writer's minimal and inconsistent control of Standard 
A merican English Usage . Non-standard usage may cause the reader to 
question the writer's understanding of standard forms. The writer's lan- 
guage may detract from the essay's effectiveness. 

The essay reveals little if any evidence of the writer's control of Standard 
American English usage . The essay may be seriously marred by the 
writer's use of non-standard forms. The writer's language may seriously 
impede the essay's effectiveness. 

The essay cannot be scor ed. YOU MUST HAVE THE TRAINER'S AUTHOR- 
IZATION TO ASSIGN THIS SCGrtE. 
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FOCUSED HOLISTIC 
SCORE SCALE 



MECHANICS DOMAIN 



The essay rev eals the writer's skilled and consistent command of the me- 
cha lies of written discourse. The essay is free of errors in capitalization, 
comma and apostrophe use, and spelling. The paragraph is indented and 
.the handwriting is clearly letfbJe. 

The essay reveals the writer's reasonable and generally consistent control 
of the mec hanics of written discourse . Although errors in capitalization, 
comma and apostrophe use, or spelling may appear, they are minor. The 
writer's intent to use paragraph format is clear, and his/her handwriting 
is reasonable. 

The ess ay reveals the writer's minimal and inconsistent control of the 
mechanics of written discourse . Errors in capitalization, comma and apos- 
trophe use, and spelling are serious. The writer's intent to format a para- 
graph may not be clear. The handwriting may be difficult to read. 

The essay reveals little if any evidence of the writer's control of the 
mechanics of written discourse . Errors in capitalization, comma and apos- 
trophe use, and spelling are obtrusive. The writer's intent to format a 
paragraph may not be clear. The handwriting may be marginally legible. 

Th e essay cannot be scored . YOU MUST HAVE THE TRAINER'S AUTHOR- 
IZATION TO ASSIGN THIS SCORE. 
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